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Abstract 


The  paper  initially  reviews  the  factors  which 
have  influenced  the  business  since  the 
1950's  and  investigates  the  trends  whieh 
have  driven  policy.  It  discusses  the  moves 
towards  a  smaller  UK  Industry  with  the 
advent  of  the  growing  affordability  issue 
and  some  of  the  measures  adopted  in 
response. 

The  review  continues  with  an  assessment  of 
the  current  international  scene  and  the 
impact  that  this  is  perceived  to  have  on 
European  Programmes  and  their  assoeiated 
Test  and  Evaluation.  From  this  base, 
consideration  is  given  to  the  future  needs  of 
testing  within  Europe,  both  in  terms  of  test 
technology  and  test  infrastructure.  A 
number  of  issues  are  raised  with  regard  to 
national  facilities,  their  affordability  and 
capability. 

The  paper  concludes  with  an  assessment  of 
these  issues  and  proposes  a  number  of 
challenges  which  need  to  be  addressed  to 
satisfy  European  Industry  needs. 

The  Historical  Perspective 

After  World  War  Two  the  major  Test  and 
Evaluation  Facilities  in  Europe  were 
Government  owned  and  managed.  Military 
aircraft  and  weapon  programmes  were 
similarly  entirely  Government  funded  and 
centrally  managed  under  cost  plus  contracts. 


The  relatively  low  cost  of  programmes 
permitted  countries  to  pursue  a  wide  range 
of  projects  simultaneously  and  several 
entirely  new  projects  could  be  started  in  a 
single  year.  In  the  second  half  of  the  1950's 
the  UK  Government  was  funding  and 
managing  the  simultaneous  development 
and  production  of  a  wide  range  of  aircraft 
programmes,  including  fighter  aircraft,  for 
the  Royal  Air  Force  and  Royal  Navy 
(Hunter,  Swift,  Javelin,  Sea  Vixen, 
Scimitar)  at  the  same  time  as  developing  the 
3  V  bombers  and  several  training,  transport 
and  general  purpose  aircraft.  A  similar 
situation  existed  in  the  helicopter  and 
missile  side  of  the  industry.  This  allowed  a 
large  number  of  relatively  small  companies 
to  survive  and  indeed  flourish,  the  cost  plus 
environment  gave  little  incentive  for  the 
companies  to  change  or  invest  in  their 
futures.  Technology  was  paramount  -  the 
threat  was  the  USSR  -  cost  did  not  seem  to 
be  a  major  issue. 

During  the  1960's  the  situation  began  to 
change  as  the  cost  of  the  technology 
required  to  achieve  combat  advantage 
escalated.  By  the  middle  of  that  decade  this 
had  forced  company  mergers  and  factory 
closures  in  the  UK  and  the  start  of  European 
collaborative  projects  such  as  the 
Anglo-French  Jaguar,  the  Franco-German 
Alpha  Jet  and  the  Tri-national  Tornado. 
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Notwithstanding  this,  European  Test 
Facilities,  continued  to  be  numerous, 
Government  owned  and  available  as  "free 
issue"  to  Contractors.  Management  of  these 
Facilities  in  the  UK  was  the  responsibility 
of  a  number  of  Government  Departments 
with  little  apparent  overall  coordination  or 
strategy.  There  was  much  duplication  of 
facilities  across  Europe,  each  country 
tending  to  own  a  total  National  capability 
which  might  vary  from  Benchmark  in  some 
facilities  to  average  or  low  in  others. 

By  the  end  of  the  seventies,  international 
collaboration  had  become  the  norm  for 
European  Industry.  Few  major  single  nation 
projects  were  being  started.  Emphasis  was 
on  establishing  common  military 
requirements  and,  through  military  and 
industrial  collaboration,  common  solutions. 

Industrial  downsizing  and  mergers 
continued  in  the  UK  such  that  there 
remained  effectively  only  one  fixed  winged 
company,  one  helicopter  manufacturer  and 
one  engine  company. 

In  the  early  1980's,  and  in  response  to  the 
trend  of  large  cost  and  timescale  overruns 
on  many  projects,  the  UK  Ministry  of 
Defence  procurement  policy  changed  from 
"cost  plus"  to  "fixed  price".  Major  new 
projects  from  the  late  1980's  e.g. 
Eurofighter,  Merlin,  Apache  and  Nimrod 
2000  as  well  as  updates  to  in-service  aircraft 
were  contracted  on  this  basis.  Furthermore 
MOD  moved  towards  "off  the  shelf 
procurement  whenever  possible  and 
employed  international  competitive 
tendering  for  many  of  the  new  requirements. 


These  changes  resulted  in  the  transfer  of  the 
risk  in  these  programmes  from  the  MOD  to 
the  Contractor,  requiring  the  Contractor  to 
manage  the  programme  and  be  responsible 
for  all  dependencies.  Consequently,  the 
approach  to  Test  and  Evaluation  changed 
with  the  Contractor  taking  full 
responsibility  for  the  overall  development 
programme.  Against  this  increasingly 
commercial  backdrop,  the  use  of  test 
methods  and  facilities  which  provided  the 
most  effective  and  timely  solution  to  the  test 
requirements  became  paramount. 

The  fall  of  the  Berlin  Wall  and  the  collapse 
of  the  former  Soviet  Union  heralded  the  era 
of  the  so  called  "Peace  Dividend"  which 
resulted  in  national  defence  budgets  falling 
almost  universally.  As  well  as  cuts  in  its 
defence  procurement  programmes,  the  UK 
Government  responded  with  moves  to 
reorganise  the  Test  Establishments  and  to 
change  fimding  policies  to  apply  more 
commercial  management  to  their 
operations.  Organisationally,  Government 
Test  Facilities  were  brought  together  under 
the  Defence  Evaluation  and  Research 
Agency  (DERA).  Consequently  DERA 
now  has  an  integrated  capability  which 
controls  all  the  Government  defence 
facilities  for  testing  of  air,  sea  and  land 
systems.  Clearly  there  are  now  many 
facilities  under  the  DERA  umbrella  and, 
whilst  inappropriate  to  list  them  all,  it  is 
considered  worthwhile  to  mention  some  of 
those  within  the  Air  Sector,  with  which  I  am 
most  familiar,  and  which  are  also 
particularly  important  to  UK  Industry. 
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There  are  several  major  instrumented 
weapon  ranges  at  Aberporth,  West  Freugh 
and  Benbecula  and  the  airfields  at  Llanbedr 
as  well  as  Boscombe  Down,  which  houses 
the  Official  Test  Centre,  FTPS  and  facilities 
such  as  the  Environmental  Test  House  and 
the  Radio  Environment  Generator.  These, 
together  with  the  Wind  Tunnels  and  Flight 
Simulators  at  Bedford  and  Famborough, 
constitute  a  comprehensive  set  of  Air  Sector 
facilities.  This  reorganisation  has  opened 
the  way,  I  believe,  for  close  collaboration 
between  Government  and  Industry  in  the 
provision  of  test  facilities  for  all  sectors  of 
Defence. 

Although  some  of  the  above  facilities 
continue  to  be  provided  free  of  charge  to  the 
Contractors  under  fixed  price  terms,  MOD 
contracts  now  generally  expect  the 
Contractor  to  provide  and  cost  for  all 
required  facilities  so  that  the  Government  is 
not  held  responsible  for  these  dependencies. 
This  in  itself  has  ensured  that  Contractors 
seek  the  most  cost  effective  solution  to  each 
requirement.  This  may  result  in  overseas 
facilities  being  used  if  appropriate  for 
example,  Eglin  AFB  in  Florida  USA  is 
being  used  for  the  Matra  BAe  ASRAAM 
trials  and  0TB  Overberg  in  South  Africa 
was  used  recently  for  Hawk  aircraft  weapon 
release  trials.  Development  of  the 
Contractors'  own  facilities  is  an  option 
where  this  is  cost  effective  in  the  longer 
term  and  joint  developments  with 
Government  are  also  possible.  The 
Electronic  Warfare  Test  Facility  (EWTF) 
completed  at  Warton  in  1997,  is  one  such 
new  National  facility  funded  equally  by 
Government  and  BAe  MA&A  and  available 
for  both  Industry  and  Government  Agency 
use. 


The  North  Sea  Air  Combat  Manoeuvring 
Instrumentation  (ACMI)  Range  is  an 
example  of  Industrial  investment  in  a 
Training  and  Test  Facility.  Owned  and 
operated  by  British  Aerospace,  but  located 
at  RAF  Bases  at  Waddington  and 
Coningsby,  it  offers  realistic  air  combat 
training  with  sophisticated  post  flight 
analysis.  Eight  European  Air  Forces  and 
the  USAFE  have  signed  contracts  and  are 
regularly  using  the  facility  which  is  the  only 
Industry  owned  and  managed  ACMI  Range 
in  the  world.  It  offers  cost  effective  training 
on  a  pay  by  hour  basis  without  the  overhead 
cost  of  ownership.  Both  DERA  Boscombe 
Down  and  BAe  MA&A  have  used  the 
facility  for  a  small  number  of 
aircrafit/weapon  system  development  flights 
and  continued  small  scale  use  is  anticipated 
on  future  programmes.  Overall,  this  type  of 
Industrial  investment,  providing  a  European 
capability,  might  prove  to  be  a  model  for 
some  future  Test  and  Evaluation  facilities. 

Moving  on  now  to  partnerships  in  the  Test 
Programmes  themselves,  Development 
Programmes  such  as  Eurofighter,  Tornado 
GR4  and  Nimrod  2000  now  include  Official 
Test  Centre  (OTC)  Pilots  and  Engineers  in 
the  Contractors  fixed  price  development 
contracts.  These  Teams  from  DERA 
Boscombe  Down  form  an  integral  part  of 
the  Company's  Test  Organisation,  carry  out 
systems  rig  and  aircraft  trials  and  fly  a 
proportion  of  test  sorties.  They  contribute 
to  test  definition  and  the  analysis  of  data 
and  have  access  to  all  test  results  as  they  are 
generated.  This  new  approach  allows  the 
OTC  to  obtain  much  earlier,  detailed  and 
continuous  experience  of  the  aircraft  and  to 
formulate  their  service  clearance 
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recommendations  progressively,  without  the 
need  for  additional  flights  and  disruption  of 
the  development  programme,  by 
transferring  test  aircraft  from  the 
Contractors  site  to  the  OTC  airfield. 

As  the  next  natural  step,  we  are  now 
drawing  the  Customer's  Operational  Team 
into  the  Development  Programme.  Nimrod 
2000  will  he  the  first  to  do  this  and  a  Joint 
Trials  Team  will  be  established  under 
Contractor  management,  which  combines 
the  talents  of  British  Aerospace,  the  Official 
Test  Centre  and  the  Royal  Air  Force 
Operators  to  ensure  the  aircraft  not  only 
satisfies  specification,  but  is  user  proven 
when  it  enters  Service. 

This  teaming  in  Contractor  Led  Trials  is 
complemented  by  the  introduction  of 
process  changes  which  also  allow  more 
effective  testing  to  be  performed.  Flight 
Test  Real  Time  Analysis  (RTA)  has  been 
introduced  by  BAe  MA&A.  This  allows 
data  transmitted  from  the  test  aircraft,  via  a 
telemetry  link,  to  be  processed  and 
compared  in  quasi  real  time  with  outputs 
from  aircraft  models  which  are  being  run  in 
parallel  on  the  ground  using  the  same  initial 
flight  conditions  as  the  test  aircraft.  Using 
this  technique  accelerated  aerodynamic 
clearance  has  been  achieved  on  the 
Eurofighter  programme  and  is  planned  for 
the  Nimrod  2000.  This  methodology  is  now 
being  expanded  to  cover  weapon  systems 
trials  by  comparing  telemetred  sensor  and 
weapons  system  data  with  data  generated 
from  sub  system  models  driven  by  reference 
data  received  in  real  time  from  the  test 
range.  Further  enhancements  are  being 
developed  through  the  use  of  IN/GPS  pods 
and  differential  correction  techniques  to 


allow  weapon  system  development  trials  to 
be  conducted  off  Range.  This  aspect  is 
dealt  with  in  more  detail  in  the  paper  being 
presented  at  this  Forum  in  Session  m. 
(Reference  1). 

Across  Europe,  and  somewhat  modestly,  the 
wider  process  of  rationalisation  of  the 
European  Defence  Industry  has  begun.  The 
Governments  of  Germany,  France  and  the 
UK  have  made  coordinated  statements  of 
intent  and  the  Industrial  Contractors  have 
responded  separately  with  their  proposals. 
Already  some  Industrial  Joint  Ventures  such 
as  Matra  BAe  Dynamics  have  been 
established  and  at  the  beginning  of  May  we 
had  the  announcement  that  SAAB  and  BAe 
would  consolidate  by  BAe  acquiring  a  35% 
stake  in  the  SAAB  Aircraft  and  missile 
business.  We  now  await  eagerly  for  more 
and  larger  groupings  to  be  announced, 
moving  towards  the  ultimate  goal  of  a  single 
European  Defence  consortium. 

Within  the  UK  a  number  of  initiatives  have 
begun  to  assess  the  Nation's  future  Test  and 
Evaluation  needs  against  this  background  of 
rationalisation.  One  such  is  the  Joint 
Ministry  Industry  Facilities  Forum  (MIFF) 
which  has  members  from  DERA,  the 
Society  of  British  Aerospace  Companies 
(SBAC)  and  the  Defence  Manufacturers 
Association  (DMA).  This  is  currently 
reviewing  capabilities  and  needs  across  a 
wide  range  of  Defence  Programmes  (Land, 
Sea  and  Air)  with  the  aim  of  identifying 
areas  where  joint  action  (rationalisation, 
harmonisation  and  co-development)  will 
benefit  the  UK's  National  T&E  Capability. 
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A  similar  European  approach  is  necessary 
and  some  degree  of  collaboration  across 
Europe  has  also  commenced.  Joint 
Government  funding  (UK,  France,  Germany 
and  Holland)  produced  the  European 
Transonic  Wind  Tunnel  in  Cologne.  This 
world  class  facility,  capable  of  full  scale 
Reynolds  number  testing,  has  been 
operational  for  3  years.  Some  National 
Wind  Tmuiels  have  been  brought  together 
and  managed  as  European  capability  under 
the  Joint  German/Dutch  Agency  DNW. 
However,  unless  accompanied  by  the  wider 
rationalisation  of  the  European  capability, 
these  initial  moves  face  potential 
commercial  failure  rather  than  success. 

There  remain  a  large  number  of  other 
National  facilities  including  Test  Ranges 
and  Test  Houses  in  Europe,  both 
Government  and  Industry  owned.  Some 
would  be  judged  to  he  excellent,  but  others 
may  be  at  best  average.  Rationalisation  of 
these  to  achieve  the  required  capacity  which 
matches  the  Defence  workload,  developing 
those  that  are  to  be  maintained,  and  closing 
or  cocooning  those  that  are  suiplus,  must  be 
a  priority.  Attention  must  be  paid  also  to 
the  way  we  manage  and  charge  for  the  use 
of  the  Facilities  -  we  must  establish 
equitable  charging  rates  and  procedures  for 
allocating  test  time  to  all  customers. 

Into  the  21st  Century 

European  defence  requirements  into  the 
next  century  will  continue  the  existing 
trends  of  regular  updates  to  in-Service 
equipments  to  prolong  their  service  lives, 
including  operational  role  changes  and 
regular  updates  to  preserve  operational 


effectiveness.  New  projects  will,  I  believe, 
focus  on  stealthy  platforms  capable  of  long 
range  engagements,  utilising  smart 
technology,  stand  off  weapons  and  other 
unmanned  vehicles.  This  will  result  in  the 
continuing  requirement  for  the  current  Test 
Ranges  and  Facilities,  however,  the  total 
volume  of  testing  will  tend  to  reduce 
following  process  improvements  and  flat  or 
even  further  reduced  National  Defence 
budgets. 

The  ongoing  changes  to  design  processes, 
which  utilise  complex  integrated  modelling 
techniques,  lend  themselves  to  "virtual 
testing"  during  the  design  phases.  These 
quite  radical  process  improvements  will 
change  our  approach  to  actual  testing  and 
will  reduce  the  number  of  physical  tests 
required.  This  process  has  already  started  in 
the  aerodynamics  discipline  where  high 
fidelity  stability  and  control,  structural  loads 
and  structural  dynamics  models  are  being 
used  in  the  flight  test  phases  of  the  most 
advanced  projects  with  significant  reduction 
in  the  amount  of  flight  testing  required. 

We  can  anticipate  that  the  development  of 
Computational  Fluid  Dynamics  (CFD)  will 
result,  in  time,  in  the  reduction  in  the  need 
for  Wind  Tunnel  testing.  It  will  also  further 
improve  the  effectiveness  of  aerodynamic 
modelling  in  general  and  this  will  also  feed 
through  into  reduced  flight  testing. 

The  ability  to  be  able  to  achieve  the 
objectives  of  traditional  between-flight 
analysis  in  quasi  real  time  has  allowed  the 
flying  programmes  to  be  carried  out  in  a 
shorter  timescale  and  resultant  lower  costs. 
Currently,  and  as  discussed  earlier,  these 
techniques  are  being  extended  to  cover  all 
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the  aircraft  flight  and  weapon  systems  and 
we  can  confidently  predict  the  same  level  of 
savings  will  be  achieved  across  the  full 
range  of  flight  testing.  The  provision  of 
"real  time"  data  from  Instrumentated 
Ranges  and  the  use  of  IN/GPS  Pods  will 
further  increase  the  rate  at  which  flight 
testing  can  be  achieved  and  has  the  potential 
to  reduce  development  programme 
timescales  and  hence  cost. 

Returning  to  Facilities  themselves,  I  believe 
that  it  is  possible  to  identify  a  number  of 
Test  Facility  shortfalls  in  the  UK  that  will 
need  to  be  addressed  over  the  next  few  years 
either  by  developing  new  or  existing 
facilities  or  deciding  to  purchase  the 
capability  overseas.  Decisions  of  this  nature 
need  to  be  taken  in  full  consultation  with  the 
relevant  Government  Agencies  and 
Industrial  organisations  if  unnecessary 
expenditure  or  duplication  is  to  be  avoided. 
Specific  capabilities  to  be  considered 
include  the  following:- 

♦  Stealth  Measurement  Facilities. 

Groimd  Radar  Cross  Section 
(RCS)  measurements  of  large 
models  and  full  scale  vehicles. 

Ground  to  Air  RCS 
measurements  of  air  vehicles 
at  all  aspects  and  including 
assessment  of  coimter 
measures. 

Air  to  Air  Infra-red  (IR) 
measurements  to  establish 
signature  versus  aspect  and 
effects  of  countermeasures  and 
decoys. 


♦  Overland  Range  for  testing  standoff 
weapons  and  unmanned  vehicles. 

♦  Land  and  Sea  Ranges  where  lasers 
can  be  accommodated  without 
undue  restrictions. 

♦  Unmanned  flying  targets  with 
realistic  performance. 

♦  Secure  data  links  from  Ranges  and 
other  major  Test  Facilities  to  allow 
test  data  transmission  to  customers 
in  real  or  quasi  real  time. 

Discussions  are  taking  place  between 
DERA,  MOD  and  Industry  on  some  of  the 
above  in  the  relevant  discipline  groups,  and 
as  mentioned  earlier,  the  recently  formed 
MIFF  has  started  reviewing  UK  capabilities 
and  requirements.  These  initiatives  need  to 
be  consolidated  within  an  overall 
framework  to  establish  the  necessary 
strategies  and  plans  to  address  all  future 
requirements.  Partnership  between 
Government  and  Industry  is  key  to  a 
successful  outcome.  I  believe  it  is  necessary 
to  achieve  a  common  understanding  of  UK 
requirements  and  capability  before  we 
engage  in  the  wider  debate  within  Europe  as 
a  whole. 

In  conclusion,  I  would  like  to  state  that  the 
discussion  of  the  reality  of  our  current  Test 
Facility  base  requires  an  inspired  look  into 
the  future,  which  is  in  itself  very  risky!  The 
future  has  a  consistent  habit  of  surprising  us 
and,  although  we  sometimes  predict  the 
direction  of  change  correctly,  we  nearly 
always  get  the  timescale  wrong! 
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Nevertheless,  I  would  like  to  leave  you  with 
the  following  conclusions  which  I  believe 
are  the  important  challenges  for  the  future:- 

Avoid  duplication  and  wasted  effort 
by  ensuring  that  Government 
Agencies  and  Industry  work  together 
in  close  partnership,  to  develop 
.  and  manage  the  Test 
Facilities  necessary  to  support  our 
Defence  programmes. 

Ensure  that  the  new  integrated 
approach  to  development  testing, 
described  as  "Contractor  Led  Trials" 
and  introduced  on  Eurofighter  and 
Nimrod  2000  is  adopted  as  best 
practice  for  Future  Projects  and  to 
extend  this  partnership  to  include  the 
Operational  Evaluation  Units 
(OEU's). 

Continue  to  develop  the  concept 
of  Real  Time  Analysis  and  to  extend 
it  to  all  major  Test  Facilities. 

Ensure  that  all  Test  and 
Evaluation  areas  are  closely 
involved  with  the  ongoing  changes 
in  design  processes  and  the 
development  of  "Virtual  Testing". 
The  opportunities  that  these 
developments  offer  to  reduce  the 
final  development  and  certification 
phases  must  not  be  wasted. 


Ensure  that,  as  the  European 
Defence  Industries  continue  the 
process  of  integration,  the  Test  and 
Evaluation  community  take 
advantage  of  the  opportunities  for 
sharing  best  practice  in  techniques 
and  methods,  establishing  centres  of 
excellence  and  rationalisation  of 
Facihties. 
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Abstract 

This  paper  describes  changes  that  are  being 
made  to  adapt  the  UK  Ministry  of  Defence  Test 
and  Evaluation  (T&E)  Ranges,  so  that  they 
meet  the  challenges  of  the  future. 

These  changes  are  being  made  to  the  way  in 
which  the  ranges  are  managed,  operated  and 
funded,  and  a  CEise  history  is  presented  to 
illustrate  the  potential  benefits  of  the  changes. 

A  clear  lesson  to  emerge  is  that  partnerships 
with  customers  and  between  National  and 
International  suppliers  of  T&E  range  services 
are  necessary  to  ensure  we  get  the  best  value 
for  money  from  our  expensive  ranges  and 
facilities. 


As  we  all  know,  the  process  of  change  is  often 
painful  and  it  is  sometimes  difficult  to  be 
confident  that  we  are  doing  the  right  thing.  I 
believe  it  can  benefit  us  all  to  share  the  lessons 
we  are  learning  as  we  each  undertake  our 
various  changes.  Perhaps,  this  paper  may  help 
some  of  you  who  are  facing  the  same  issues  as 
we  are  in  the  UK  T&E  ranges,  and  conversely, 
if  any  one  feels  we  are  doing  it  wrong  then  you 
will  tell  us  -  sooner  rather  than  later  would  be 
appreciated! 

DERA  T&E  Ranges 

The  UK  has  many  T&E  ranges,  generally 
situated  in  out-of-the-way  places  for  safety  and 
security  reasons,  but  as  a  consequence  also  in 
some  of  the  most  beautiful  areas  of  the  country. 


Introduction 

The  only  certainty  about  the  future  is  that 
everything  will  continue  to  change,  and  that  the 
pace  of  change  is  accelerating.  These  changes 
are  likely  to  involve: - 

-  the  threat; 

-  the  nature  of  conflict; 

-  technologies; 

-  computer  based  modelling  and  simulation; 

-  budget  priority; 

-  procurement  processes; 

-  multi-national  programmes; 

-  globalisation. 
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DERA  T&E  ranges 


At  the  moment  we  can  only  dream  about  the 
directions  in  which  some  of  these  changes  are 
going  to  take  T&E  over  the  next  few  years,  and 
yet  we  have  to  prepare  for  the  future. 


These  ranges  provide  safe  and  secure  spaces 
and  facilities  for  inherently  dangerous  tests, 
experiments,  trials  and  exercises  in  all  three 
environments  -  land,  sea  and  air. 
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Land  ranges  are  used  for  work  spanning 
research  and  development  through  to  in- 
service  proof  of  artillery  systems  and  small 
and  medium  calibre  weapons,  including 
environmental  testing  and  disposal  of 
munitions,  and  for  static  and  dynamic 
work  requiring  large  danger  areas; 

Sea  ranges  are  used  primarily  for 
measurements  of  acoustic  signatures, 
magnetic  signatures  and  for  underwater 
tracking  and  system  calibration  work; 

Air  ranges  are  used  for  test  and  training 
firings  of  all  variants  of  guided  and 
unguided  missiles,  UMA  and  air  delivery 
systems. 

Until  1995  the  UK  ranges  where  treated  as 
relatively  autonomous  establishments,  although 
many  were  collectively  grouped  in  an 
organisation  called  DGTE  (  Director  General 
Test  and  Evaluation)  by  this  time. 

In  1995  DGTE  joined  forces  with  3  other  parts 
of  the  UK  MoD  to  form  the  Government 
owned  Defence  Evaluation  and  Research 
Agency,  DERA.  Within  DERA  all  the  T&E 
ranges  have  now  been  grouped  into  a  single 
sector  which  has  a  turnover  of  about  £120m  per 
year. 

Funding  arrangements 

When  the  ranges  joined  DERA  in  1995  they 
were  transferred  from  vote  fimd  to  a  trading 
fund  accounting  system. 

I  believe  that  this  change  will  prove  to  be  the 
single  most  important  event  that  will  stimulate 
and  allow  facility-rich  organisations  such  as  the 
T&E  ranges,  to  adapt  to  the  rapidly  changing 
world. 

Under  the  vote  system.  Parliament  through  the 
Treasury  allocates  money  to  MoD  each  year, 
and  this  allocation  is  further  sub-allocated  by 
MoD  HQ  to  different  departments  and  so  on 
down  the  management  chain. 


Prior  to  1995,  each  of  the  functions  on  the  T&E 
ranges  received  their  share  of  this  allocation  to 
fund  their  activities. 


Pre-Agency  funding  arrangements 


Under  vote  funding,  the  ranges  developed  some 
very  distinct  organisational  features  such  as:- 

-  a  hierarchical  management  structure  with 
loyalty  and  responsiveness  up  the 
management  chain  of  command  (because 
that  is  where  the  money  comes  from); 

-  departmentalised  or  “stove  piped” 
functions,  with  each  department  having  its 
own  priorities.  To  get  a  job  done  that 
required  several  departments  was  difficult 
to  co-ordinate,  since  departments  only 
came  under  a  common  manager 
somewhere  high  in  the  MoD  HQ 
organisation; 

-  a  static  organisation,  since  the  vote  in  any 
one  year  was  generally  much  the  same  as 
for  the  previous  year,  and  there  were 
relatively  long  lead  times  for  major 
changes.  This  made  the  organisation  slow 
to  respond  to  change; 

-  long  lead  times  in  getting  major  new 
investment  to  keep  facilities  up  to  date; 

-  each  site  being  self-sufficient  to  cope  with 
peak  loads,  leading  to  duplication  of 
capability  and  excess  capacity  across  the 
ranges. 
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In  1995  this  situation  changed,  and  many  MoD 
facilities,  including  the  ranges,  joined  DERA 
under  a  trading  fund  arrangement  as  illustrated 
in  the  next  figure. 
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DERA  funding  arrangements 

Money  is  no  longer  allocated  each  year  to  the 
ranges,  but  the  ranges  share  of  the  vote  is  given 
to  the  users  or  ‘customers’  of  the  ranges. 

Consequently,  the  ranges  now  only  get  money 
to  sustain  their  operation  when  they  carry  out 
tasks  that  customers  request.  The  trading  fund 
requires  us  to  recover  all  costs  of  operation  plus 
a  small  return  to  Treasury  (but  I  hasten  to  add 
we  are  not  a  profit  making  organisation). 

Also,  because  the  customers  now  have  the 
money,  they  are  no  longer  bound  to  use  the 
T&E  ranges. 

This,  as  all  of  you  from  industry  have  known 
for  a  long  time,  concentrates  the  mind  on 
understanding  customer  requirements  and 
customer  service  -  especially  when,  as  in  the 
case  of  ranges,  the  customers  think  the  service 
is  much  too  expensive  now  they  have  to  pay  for 
it  rather  than  get  it  for  free. 

An  immediate  reaction  to  this  situation  was  to 
question  whether  the  ranges  could  survive. 
Now  it  is  said  that  a  crisis  is  needed  to  motive 
change  and  in  a  way  a  crisis  has  been  created. 


ways  for  managing  and  operating  our 
(expensive)  facilities,  as  outlined  below. 

The  management  of  UK  T&E  ranges  since 
1995 

In  making  the  transition  to  a  trading  fund  it 
quickly  became  apparent  that  the  “stove  pipe” 
type  management  was  very  inappropriate.  The 
organisational  responsiveness  had  to  be  re¬ 
focused  to  where  the  money  now  comes  from  - 
ie  the  customer.  To  achieve  this  within  the 
ranges  sector  we  have  adopted  the  DERA 
project  management  based  approach  to  react  to 
and  solve  customer  needs  and  problems. 

For  each  customer  requirement,  we  set  up  a 
project  to  interpret  the  customer’s  needs,  to 
develop  solutions  to  meet  that  need,  and  to 
marshal  and  manage  all  the  resources  needed  to 
deliver  what  the  customer  wants.  This  means 
drawing  on  the  pool  of  Human  Resources  to 
build  the  necessary  multi-functional  teams  to 
undertake  the  T&E  work,  and  the  facilities 
needed  for  the  particular  job  in  hand.  When  we 
have  delivered  our  service  or  T&E  results  to 
our  customers’  satisfaction,  we  then  get  paid 
for  the  job. 


Customer  focused  organisation 

The  immediate  advantages  of  the  new 
arrangements  are:- 

-  greater  responsiveness  to  customer  needs; 

-  easier  investment  processes  to  enhance 
facilities  by  borrowing  from  the  trading 


In  the  case  of  T&E  ranges  it  has  induced  us 
actively  to  seek  and  implement  innovative 
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fund  (assuming  a  good  business  case  is 
made)  since  costs  of  the  investment  can  be 
recovered  steadily  from  customer  over  a 
number  of  years  rather  than  in  one  large 
payment; 

-  areas  or  resources  not  in  demand  or  used  to 
solve  customer  requirements  are  quickly 
identified  because  they  do  not  recover  their 
costs,  and  necessary  action  can  be  put  in 
place  to  correct  for  this. 

This  final  point  was  at  first  thought  to  be  a 
major  stumbling  block,  when  the  implications 
of  the  trading  fund  arrangement  became 
apparent.  With  the  loss  of  stability  of  the  vote 
funding,  the  question  was  asked  how  could 
facilities  be  sustained  when  the  demand  (and 
hence  volume  of  work)  for  them  varies 
considerably  -  perhaps  over  a  several  year 
cycle?  How  can  facilities  recover  their  costs  in 
the  lean  times? 

For  ranges  the  solution  lies  in  not  treating  each 
facility  or  site  as  an  individual  self  contained 
entity,  but  applying  the  new  management 
model  across  all  ranges.  For  example:- 

-  the  staff  management  is  networked  across 
the  ranges  so  that  people  from  one  range 
having  a  quiet  time  can  help  out  at  ranges 
which  are  busy  -  thus  avoiding  each  range 
catering  for  peak  loads  individually.  This 
implies  much  more  multi-skilling  of  staff 
(to  turn  their  hands  to  many  different 
tasks)  and  greater  flexibility  of 
deployment,  which  in  turn  requires 
training  investment  and  better  rewards  for 
people; 


All  these  measures  can  help  reduce  the  costs  of 
operating  facilities  and  offer  better  value  for 
money. 

In  the  new  management  structure  we  now  have 
to  set  a  rate  for  the  job  so  that  the  payments  we 
receive  cover  not  only  the  workers  undertaking 
the  T&E,  but  also  cover  associated  central 
management  costs  and  all  infra-structure  costs. 
For  ranges  sector  these  costs  represent  5%  and 
46%  of  total  costs  respectively. 


directors  5%  of  costs 


site  management  Security  Fire  services  infra-structure 
accommodation  maintenance  utiiities  4@i)/,  of  costs 


Full  costs  of  service 

The  role  of  central  management  is  no  longer 
one  of  control,  but  is  much  more  concerned 
with  support  and  setting  the  strategy  for  the 
future  -  it  is  all  about  creating  an  environment 
conducive  to  effective  and  efficient  range  T&E 
work. 

The  infra-structure  costs  -  the  security,  health 
and  safety,  accommodation,  utilities,  site  costs 
and  liabilities  -  are  traditionally  regarded  as 
fixed  costs;  they  are  incurred  whether  or  not  we 
carry  out  any  customer  work  using  the 
facilities. 


sharing  of  facilities  across  ranges, 
especially  perhaps  mobile  instrumentation, 
which  helps  remove  duplication  of 
capability  and  ensures  higher  utilisation  of 
assets; 


So  really,  it  could  be  argued  that  we  can  only 
affect  49%  of  the  costs  (ie  doing  the  customer’s 
work)  by  the  new  flexible  management 
arrangements. 


not  tying  project  management  to  a  given 
range  but  encouraging  project  managers  to 
look  across  all  ranges  for  the  best  solutions 
to  customer  problems. 


But  this  is  not  so,  and  partnering  arrangements 
can  provide  the  key  to  reducing  the  infra¬ 
structure  costs. 


5-4 


Partnerships 


A  lot  of  the  work  needed  to  maintain  our  site 
and  facility  infra-structure  is  currently 
subcontracted  on  traditional  MoD  customer  - 
supplier  type  contracts.  For  the  future  we  are 
striving  to  move  to  partnership  arrangements, 
in  which  we  have  a  more  equitable  share  of 
risks  in  return  for  a  more  equitable  share  of 
benefits. 

The  benefit  of  partnering  arrangements  is  that 
we  work  together  more  flexibly  to  solve 
problems  and  accrue  benefits  that  we  both  can 
share,  for  example,  by  exploiting  the  ranges 
and  facilities  for  uses  other  than  their  primary 
purpose,  or  by  only  switching  on  the  facility 
when  it  is  needed  for  use  rather  than  keeping  it 
ready  for  action  at  all  times.  One  might  even 
float  the  idea  of  ‘T&E  parks’  akin  to  the  more 
traditional  ‘science  parks’. 

We  also  need  to  work  with  other  suppliers  of 
facility  based  T&E  both  nationally  and 
internationally,  who  are  subject  to  the  same 
issues  as  we  are.  We  need  to  develop  ways  of 
harmonising  our  management  and  sharing  of 
facilities,  and  this  has  to  be  much  more  than  the 
occasional  contract  to  use  another  supplier’s 
facilities.  It  will  require  a  meeting  of  cultures 
and  minds,  and  will  take  a  determined  effort 
and  time  to  really  get  it  set  up  -  but  the  benefits 
will  be  very  worthwhile. 

We  also  need  to  develop  improved  partnerships 
with  our  customers,  so  that  we  ensure  that  they 
get  the  best  use  of  T&E  facilities  to  help  solve 
their  problems  and  reduce  their  risks. 

A  lot  of  investment  is  needed  to  support  the 
necessary  cultural  changes 

-  to  introduce  campaign  teams  and  multi¬ 
skilling; 

-  to  develop  effective  project,  resource  and 
facility  managers  who  work  together  as  a 
team  and  who  network  across  facilities  and 
ranges; 

-  to  provide  full  empowerment  of  the  local 


teams  to  do  the  work  and  make  changes  for 
improvement; 

-  to  develop  trusting  relationships  for  range 
support  and  exploitation; 

-  to  establish  arrangements  with  other 
national  and  international  suppliers  of 
range  based  T&E. 

Theory  into  practice 

I  would  now  like  to  present  a  case  history  to 
illustrate  the  potential  of  the  management 
methods  I  have  just  outlined. 

The  ranges  sector  is  in  the  throes  of  defining 
the  future  of  3  of  its  land  ranges.  For  a  number 
of  reasons  the  customers’  needs  to  use  these 
ranges  have  declined  and  the  ranges,  because  of 
their  history,  have  duplication  of  capability  and 
excess  capacity. 

The  following  figure  shows  the  level  of  T&E 
work  we  expect  for  the  3  ranges,  measured  in 
terms  of  projected  customer  income  over  the 
period  1996  to  2001.  Under  the  trading  fund  we 
now  collect  all  the  costs  of  operating  the 
ranges,  and  this  showed  that  a  required  income 
of  £40m  pa  was  needed  when  the  ranges  joined 
DERA  in  1995,  to  meet  the  requirements  of  the 
trading  fund  (ie  cover  costs  plus  standard  return 
to  Treasury). 


Income  requirements  -  T&E  land  ranges 

By  simply  squeezing  costs  and  curtailing  all 
investments,  the  required  income  to  cover  the 
costs  of  the  ranges  can  be  reduced  to  about 
£3  5m  per  year.  However,  as  can  be  seen  from 
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the  above  figure,  this  is  nowhere  near  enough 
to  make  the  ranges  viable  in  the  face  of  the 
falling  customer  demand. 

By  implementing  some  of  the  fundamental 
changes  outlined  above,  the  required  income 
can  be  reduced  to  about  £22m  per  year,  whilst 
still  retaining  investments  for  the  future,  and 
this  is  much  more  in  line  with  projected 
customer  income.  I  also  believe  that  our 
customers  will  find  that  we  will  become  a 
much  more  focused  and  responsive 
organisation.  I  would  add  at  this  stage  that  we 
are  not  there  yet,  as  we  are  now  in  the  middle 
of  making  these  changes.  I  hope  I  can  report 
success  at  the  next  conference  in  2  year’s  time! 

Evolving  nature  of  T«feE 

As  we  try  to  predict  what  the  future  holds  for 
T&E  ranges  and  facilities,  the  key  factors  to  be 
considered  are:- 

-  how  will  the  defence  equipment 

procurement  process  develop  and  what 
will  customers  require  by  way  of  T&E  in 
this  process; 

-  what  systems  (or  technologies)  are  being 
developed  which  will  require  T&E  results 
and  services; 

-  and  what  new  technologies  can  be  used  in 
our  facilities  to  provide  better  results  and 
knowledge  capture  to  meet  future  T&E 
requirements? 

The  process  for  defence  equipment 
procurement  in  the  UK  is  currently  under  close 
examination  as  part  of  the  Strategic  Defence 
Review,  and  whilst  details  of  any  alternative 
approaches  to  current  procurement  practices  are 
not  yet  known,  it  is  likely  that  a  controlled 
evolutionary  approach  to  upgrading  operational 
capabilities  will  come  into  play. 

Factors  that  will  gain  in  significance  are:- 

-  increased  use  of  modelling,  stimulation, 
simulation  and  synthetic  environments  to 
evaluate  options  for  enhancing  capabilities, 
provide  training  environments,  and  to 


provide  reference  and  specifications  for 
accepting  new  equipment; 

-  international  projects  for  equipment 
procurement; 

-  technology  insertion,  including  the  use  of 
off  the  shelf  technology  /  systems; 

-  shortening  of  procurement  timescales; 

-  the  emerging  paradigm  of  systems 
engineering  to  provide  a  framework  for  the 
whole  process  of  capability  enhancement. 

A  representation  of  systems  engineering  is 
illustrated  in  the  following  figure. 
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Systems  engineering 


Operational  capability  is  provided  by  the 
totality  of  systems  (including  people  and  the 
operational  procedures  and  equipment  they 
use)  and  this  “system  of  systems”  can  be 
broken  down  into  the  individual  systems  and 
then  individual  system  components  which 
ultimately  depend  on  the  available 
underpinning  science  and  technology.  Systems 
engineering  provides  the  discipline  for  the 
specification,  control,  acceptance  and 
traceability  of  changes  made  to  the  system 
configuration. 

During  a  particular  system  procurement,  the 
options  can  be  evaluated  at  various  levels  from 
“system  of  systems”  down  to,  perhaps,  the 
component  level  to  determine  how 
enhancements  to  operational  capability  can  be 
made. 
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This  is  an  interactive  process,  and  once  a 
preferred  option  begins  to  crystallise,  then 
specifications  for  its  acceptance  must  be 


established  -  so  when  the  change  is  made  we 
ean  measure  that  it  has  met  its  success  criteria. 

It  is  to  be  hoped  that  by  undertaking  the  option 
evaluation  correctly,  the  process  of  synthesis  of 
the  preferred  option  and  its  eventual  integration 
into  the  system  of  systems  becomes  a  low  risk 
exercise,  and  T&E  has  a  signifieant  part  to  play 
in  this  process. 

During  requirements  definition,  T«&:E  will  be 
used  to  collect  fundamental  real  world 
information  for  input  into  the  models  use  for 
evaluating  options.  For  ranges  this  means 
providing  a  safe  space  to  conduct  large  scale 
experiments  and  measurements  on  various 
system  components  and  technologies.  Also  at 
this  stage,  the  methods  and  then  the 
complementary  specifications  for  acceptance  of 
the  options  selected  for  implementation  must 
be  compiled. 

Once  we  move  into  the  development  and 
integration  of  components  and  systems,  then 
verification  is  necessary. 

Whilst  modelling  and  stimulation  will 
undoubtedly  play  a  much  larger  role  than  in  the 
past  for  acceptance  evaluation,  there  will  still 
be  a  requirement  for  testing  at  the  component 
level,  and  “system  of  systems”  testing  at  the 
highest  levels  where  T&E  will  be  much  richer  - 
perhaps  including  operational  evaluation  with 
operators,  operational  procedures  and  training. 

The  results  of  the  “system  of  systems”  level 
T&E  is  then  feedback  to  the  assessment  of 
needs  for  the  next  phase  of  enhancing 
capability. 

In  this  model,  T&E  using  range  facilities  can 
now  be  seen  as  supporting  all  phases  of  a 
process  of  incremental  “improvement  of 
capability”  -  and  is  in  fact  being  used  for 
through-life  risk  reduction. 

The  second  factor  that  will  drive  the  future  of 
facilities  (retention  or  enhancements)  is  clearly 
the  types  of  system  and  equipment  that  have  to 


be  measured.  Trends  for  the  future  may  well 
be:- 

-  a  move  to  autonomous  systems  -  a  step 
beyond  smart  weapons  -  made  possible  by 
miniaturisation  of  powerful  processors  and 
sophisticated  sensors; 

-  systems  that  can  influence  wider 
geographical  areas  -  long  range  weapons, 
wide  area  sensors  (and  including  command 
and  control); 

-  more  off  the  shelf  systems; 

-  influence  of  counter-measures; 

-  ‘smart’  materials  which  adapt  to  the 
environment; 

-  non-lethal  weapons; 

-  UAVs; 

-  stealth  technology; 

-  and  more  interest  in  the  performance  at  the 
system  integration  level  rather  than  just  the 
component  test  level. 

The  third  factor  affecting  the  future  of  the 
ranges  is  to  provide  the  capability  to  meet  the 
above  requirements  for  measurement. 

This  will  require  significant  investment  in  new 
and  upgraded  facilities  that  are  already  deemed 
to  be  expensive! 

The  future 

And  so  to  the  future! 

Taking  into  account  the  factors  outlined  above, 
I  believe  that  working  together  on  a  National 
and  International  basis  is  the  only  viable  way 
forward  for  T&E  ranges  and  facilities. 

Sharing  expensive  facilities  that  are  only  used 
occasionally  makes  sense,  and  mirrors  the 
move  towards  multi-national  equipment 
procurement  programs. 

However,  to  work  together  we  need  to  have  a 
common  vision  of  where  we  are  going  and  how 
T&E  range  facilities  might  fit  into  the  future. 
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There  are  many  ways  of  looking  at  this,  and  I 
put  forward  the  following  diagram  representing 
what  we  should  be  striving  towards  -  namely, 
closer  integration  of  results  from  real  world 
measurements  with  the  models  that  support  the 
development  and  implementation  of  enhanced 
operational  capability. 
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role  of  test  and  evaluation 

I  would  welcome  views  and  suggestions  on 
how  we  might  do  more  together,  to  define  a 
common  vision  for  the  role  of  T&E  ranges,  and 
to  take  the  necessary  steps  to  realise  it. 

Thank  you. 
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SELLING  CHANGE  IN  THE  “NEW”  BOEING  COMPANY: 
INTEGRATING  600  LABORATORIES 
William  E.  Newman 
Richard  J.  Jablonski 
Phantom  Works 

Boeing  Information,  Space  &  Defense  Systems  Group 
Seattle,  WA  USA 


Abstract 

The  Information,  Space  and  Defense 
Systems  (ISDS)  Group  of  The  “new” 
Boeing  Company  bears  slight 
resemblance  to  the  former  Boeing 
Defense  and  Space  Group.  The  merger 
of  Boeing  with  Rockwell  and  then 
McDonnell  Douglas  created  a  complex 
marriage  of  different  product  lines, 
cultures,  accounting  systems,  personnel 
systems,  and  management  styles.  Our 
differences  became  particularly  obvious 
when  we  considered  the  options  of  how 
to  integrate  the  heritage  management 
philosophies  and  operating  systems  for 
the  combined  600  Research, 
Development,  Test  &  Evaluation 
(RDT&E)  laboratories  and  test  facilities 
of  these  three  companies.  A 
fundamental  question  was  how  to 
strategically  integrate  and  manage 
similar  yet  widely  separated  lab 
resources,  operate  them  as  a  business  in 
order  to  maximize  operational 
efficiency,  and  yet  optimize  support  to 
customers,  regardless  of  lab  location. 

In  November  1997  the  Information, 
Space  and  Defense  Systems  Group 
created  Enterprise  Laboratories,  a  new 
Group-wide  laboratory  management 
organization  under  Phantom  Works,  one 
of  the  four  business  units  of  the  new 
Group.  Chartered  to  make  strategic 
consolidation  recommendations  as  their 
initial  thrust,  Enterprise  Laboratories 
was  the  Group’s  first  organizational 
element  to  operate  across  all  sites  and 


business  units.  Their  goal  was  to  find 
new  efficiencies  and  attendant  savings 
through  the  consolidation  of  laboratory 
assets  across  the  former  companies.  To 
reverse  an  old  adage,  the  sum  should  be 
less  than  the  parts.  Three  months  later 
Enterprise  Laboratories  presented  145 
recommendations,  nearly  equally  divided 
between  closures  and  consolidations, 
totaling  $100  million  in  5-year  gross 
savings.  Group  leadership  approved 
these  recommendations  in  March  1998. 
Along  with  this  approval,  they  also 
chartered  a  new  management  philosophy 
for  these  600  general  usage  labs  and 
renamed  the  organization  ISDS 
Laboratories. 

The  tasks  of  ISDS  Laboratories  now 
center  on  delivering  this  first  round  of 
identified  savings  while  finding  ways  to 
provide  laboratory  managers  and  their 
customers  additional  increased 
efficiencies.  To  achieve  this  goal  the 
existing  management  of  the  heritage 
companies  must  also  take  ownership  of 
the  change  process.  In  short,  all 
management  levels  must  believe  that  the 
end  state  will  benefit  their  customers, 
their  people,  and  themselves.  The  new 
voyage  has  begun  but  is  far  from  over. 

The  Mergers 

Following  the  approval  the  Boeing 
merger  with  Rockwell’s  similar  defense 
and  space  interests,  we  commenced  a 
study  in  January  1997  to  integrate  the 
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Boeing  defense  and  space  laboratories 
with  like  facilities  of  our  new  Rockwell 
teammates  in  southern  California.  This 
5-month  study  culminated  in  1 16 
recommendations  for  closure  and 
consolidation.  The  integration  approach 
was  to  first  gain  understanding  of  the 
capabilities,  equipment  condition,  and 
business  situation  for  all  Boeing  and 
Rockwell  labs  by  using  a  “inter¬ 
company”  integration  team  of  experts  to 
evaluate  consistent  data  in  conjunction 
with  actual  tours  of  sites  and  labs.  A 
standard  laboratory  data  sheet  was 
created  to  capture  these  physical, 
operational,  and  business  aspects  for  an 
individual  facility.  The  lab  managers  at 
all  sites  then  completed  data  sheets  for 
each  lab  and  test  facility.  These 
laboratory  data  sheets  have  become  the 
core  of  an  electronic  relational  data  base 
that  provides  the  common  denominator 
for  strategically  managing  laboratories 
across  the  Information,  Space  and 
Defense  Systems  Group. 

The  consolidation  recommendations 
submitted  in  June  1997  were  the 
consensus  of  this  “inter-company”  team. 
However,  difficulties  were  encountered 
in  implementing  the  recommendations 
because  of  three  main  reasons:  (1)  the 
lack  of  a  senior  lab  manager  empowered 
to  lead  the  implementation  phase,  (2) 
the  lack  of  a  company-wide  finance 
mechanism  to  absorb  (or  at  least 
absolve)  the  non  budgeted 
implementation  costs,  and  (3)  buy-in  by 
the  affected  middle  managers  of  the 
heritage  companies.  These  realities 
further  abetted  the  understandable 
human  nature  to  protect  one’s  immediate 
organization  from  upper  level 
management  decisions,  which  upset  the 
local  status  quo.  These  were  important 
lessons  learned  for  the  next  merger  effort 
with  McDonnell  Douglas. 


July  1997  brought  the  stockholders’ 
approval  of  the  merger  with  McDonnell 
Douglas,  and  this  had  the  greatest  impact 
on  the  defense  and  space  programs  of 
The  (new)  Boeing  Company,  starting 
with  the  title:  Information,  Space  and 
Defense  Systems  (ISDS)  Group.  The 
Group  is  comprised  of  four  operating 
business  units:  Space  Transportation, 
primarily  located  in  Southern  California, 
Aircraft  and  Missile  Systems,  primarily 
located  in  St.  Louis,  Information  and 
Communication  Systems,  primarily 
located  in  Seattle,  and  Phantom  Works, 
the  advanced  technology  and  cross-site 
enabling  network  which  supports  the 
other  Group  business  units  at  all 
operating  sites.  Hence,  it  was  decided  to 
place  the  management  structure  for 
hundreds  of  labs  across  the  nation  in  the 
Phantom  Works  organization. 

Enterprise  Laboratories  became  an 
official  element  of  Phantom  Works  in 
November  1997.  From  the  outset,  there 
were  two  central  tenets  of  this  new 
organization:  (1)  to  base  integration  and 
other  major  lab/test  facility  decisions  on 
detailed  business  case  analyses 
supported  by  accurate  data,  and  (2)  to 
centrally  coordinate  strategic 
management  actions  for  similar  facilities 
across  the  business  units  and  sites.  The 
issue  of  managing  daily  lab  operations 
will  be  discussed  later  in  this  paper. 

The  Process 

The  modus  operandi  for  the  new 
organization  was  to  first  group  all 
general  usage  labs,  regardless  of 
location,  into  a  manageable  number  of 
categories  with  similar  technical  and 
operational  requirements.  The  eight 
categories  selected  were:  Aerodynamics 
&  Stmctures,  Electronics, 
Signature/EMFEMC,  Modeling  & 
Simulation/  Systems  Development/MMI, 
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Propulsion/Physics/Fluid  Mechanics, 
Environmental/Mechanical,  Materials  & 
Processes,  and  Metrology.  At  the  outset, 
there  were  approximately  600  labs 
among  these  categories.  (It  must  be 
noted  here  that  within  the  Group  there 
are  an  additional  160  labs,  not  under 
Enterprise  Laboratories  management, 
which  are  dedicated  to,  managed,  and 
directly  funded  by  individual  programs, 
and  these  are  typically  systems 
integration  labs.)  Each  lab  technical 
category  was  assigned  to  a  Laboratory 
Strategic  Manager  (LSM)  who  had 
expertise  in  the  required  disciplines,  and 
these  managers  were  assigned  from 
laboratory  management  positions  within 
the  former  heritage  companies.  These 
eight  technical  groupings  permitted  a 
reasonable  number  of  labs  for  the 
assigned  leaders  to  manage. 

The  next  step  was  to  determine  the 
schedule  and  process  for  creating  a  first 
round  of  integration  recommendations. 
With  the  Enterprise  Labs  team  now 
comprised  of  people  from  the  three 
former  companies  and  their  major 
geographical  regions,  it  became  obvious 
that  the  LSMs  would  have  to  personally 
visit  and  review  all  600  general  usage 
labs.  Despite  a  larger  number  of  labs  to 
assess,  it  was  decided  to  reduce  the 
schedule  to  three  months  by  having  each 
category  of  labs  visited  only  by  the  LSM 
having  cognizance  for  that  category. 
They  set  their  own  travel  schedule  for 
gathering  data  and  traveled  alone  or  with 
a  very  small  staff.  Their  primary 
objectives  were  to  validate  the 
information  provided  on  the  laboratory 
data  sheets,  brief  local  laboratory 
managers  on  the  consolidation  process, 
and  develop  a  detailed  contact  list  for 
future  strategic  planning. 

The  team  maintained  a  very  fast  pace 
during  the  site  visit-data  gathering  phase. 


Two  interim  situation  reports  to  the 
executive  management  of  Phantom 
Works  punctuated  this  three-month 
phase.  During  these  briefings  the  team 
received  guidance  on  the  changing 
terrain  of  the  merger  and  where  to  focus 
their  efforts.  In  the  final  month  they 
created  business  cases  for  each 
recommendation  based  on  the  assessed 
data,  and  a  criterion  was  established  for 
a  closure/consolidation  decision.  All 
business  unit  stakeholders  attended  and 
critiqued  the  final  presentation  of 
recommendations.  The  results  are 
covered  in  a  later  section  of  this  paper. 

Laboratory  Ownership 

From  the  start  of  the  Enterprise 
Laboratories  integration  study,  the 
concept  of  “centralized  decision 
authority”  over  all  ISDS  general  usage 
labs,  for  whatever  purpose,  caused  lively 
debates  regarding  lab  ownership,  and 
who  had  what  authority  over  these  key 
engineering  resources.  First,  the  new 
Group  business  units,  created  from  the 
heritage  companies,  still  maintained  the 
overhead  budget  “pools”  and  the 
numerous  financial  accounting  systems. 
This  situation  remained  virtually 
unchanged  after  the  mergers,  and  major 
changes  to  these  basic  business  systems 
may  require  years.  Second,  the  strong 
preeminence  of  program  management 
teams  in  the  heritage  companies 
generated  an  understandable  suspicion  of 
any  new  “central”  organization  claiming 
to  have  authority  over  their  key 
engineering  laboratories  and  test 
facilities.  Finally,  there  will  always  he 
middle  management  resistance  to  change 
in  large  organizations,  even  when 
significant  events  like  nation-wide 
mergers  clearly  indicate  the  need  for  it. 
The  merger  of  three  large,  distinct 
cultures  resulting  in  a  new  company  of 
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235,000  people  certainly  provides  fertile 
ground  for  the  growth  of  such 
disagreement. 

Management  of  Daily  Operations 

Hundreds  of  full-time  “overhead” 
personnel  are  required  to  maintain  and 
operate  the  600  general  usage  Group 
laboratories.  A  central  assumption  from 
the  beginning  was  that  the  day-to-day 
support  of  programs  would  eventually  be 
managed  under  Enterprise  Laboratories. 
Obviously,  if  this  happened,  large 
portions  of  the  existing  lab  management 
organizations  at  all  sites  would  have  to 
become  part  of  Enterprise  Labs’  line 
management  authority.  In  the  early 
months  following  creation  of  Enterprise 
Laboratories,  this  transformation  was 
discussed  only  in  very  broad  terms, 
adding  further  concerns  to  those  of  the 
ownership  issue. 

The  Results 

The  results  of  the  team’s  efforts  to  date 
are  gratifying.  At  the  end  of  the  three 
month  effort,  they  recommended  76 
laboratories  for  closure  and  69 
laboratories  for  consolidation.  When 
implemented,  these  actions  will  result  in 
savings  to  The  Boeing  Company 
approaching  $100  million  over  five 
years.  The  primary  contributors  are 
operating  costs  in  the  form  of  square 
footage  and  employee  reductions,  plus 
cost  avoidance  for  future  maintenance 
and  capital  improvements. 

The  Laboratory  Strategic  Managers  have 
established  personal  contact  with  the 
individual  managers  for  each  laboratory 
in  their  category.  These  contacts, 
combined  with  their  understanding  of  the 
lab  capabilities  and  assets,  enables  the 


LSMs  to  plan  strategic  investments  and 
resource  allocations.  Their  perspective 
of  viewing  laboratory  capabilities  and 
needs  across  the  Group,  independent  of 
location  and  heritage  company 
affiliation,  assures  strategic  decisions  in 
the  best  interests  of  Boeing. 

An  expanded  Laboratory  Data  Base  now 
includes  increased  levels  of  detail  and  is 
available  to  all  affected  Boeing 
personnel.  The  relational  data  base  tools 
provided  with  the  software  application 
allow  users  to  sort  and  format  the 
information  according  to  their  needs. 
Information  on  laboratory  capabilities  is 
available  to  all  internal  customers  for 
their  use  in  proposal  planning  and 
resource  identification.  The  data  base 
will  be  available  on  a  web  server  by  mid 
1998  thereby  enabling  easier  access  for 
users  and  customers.  Further  expansion 
will  include  capital  and  Internal 
Research  &  Development  (IR&D) 
planning  information. 

The  Future 

As  the  Information,  Space,  and  Defense 
Systems  Group  approved  these 
recommendations,  they  also  introduced  a 
new  laboratory  management  philosophy 
and  structure.  First,  the  title  was 
changed  to  ISDS  Laboratories,  reflecting 
the  Group- wide  scope  of  strategic  lab 
management.  Second,  ISDS 
Laboratories  received  a  clear  mandate  to 
perform  these  strategic  management 
functions  for  all  600  general  usage  labs 
across  ISDS.  The  day-to-day 
management  of  projects  will  continue  to 
reside  within  the  regional  site 
management  structures.  However,  four 
of  the  team’s  eight  lab  category 
managers  are  also  “dual  hatted”  as 
Regional  Site  Managers.  This  new 
feature  provides  close  connectivity 
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between  the  strategic  and  daily 
management  roles.  The  team  is 
organizing  the  next  round  of  laboratory 
integration  recommendations.  Strategic 
management  functions  such  as  major 
resource  allocation  (capital  investments, 
IR&D)  and  major  lab  process 
improvements  (e.g.,  general  purpose  test 


equipment  management)  also  became 
the  purview  of  ISDS  Laboratories  with 
this  modified  organizational  structure. 
Boeing  continues  to  seek  ways  of 
improving  shareholder  value,  and 
efficient  laboratory  and  test  facility 
management  structure  will  greatly 
contribute  to  our  return  on  net  assets. 
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Rafael 's  WTEC  has  grown  together  with  weaponry  development  in  Israel  In  the  last  deccule, 
the  Israeli  MOD  budget  for  armament  development  has  been  reduced  which  implied  less  new 
armament  developments.  This  result  in  fewer  test  missions  to  be  performed  at  WTEC. 

This  article  presents  WTEC  experience  and  strategy  to  face  the  challenge  of  being  more 
efficient  in  perfonning  the  T&E  mission.  This  goal  is  achieved  by  enhancing  the  productivity 


and  reengineering  the  T&E  process. 

Rafael  preseiitation 

RAFAEL,  the  Armament  Development 
Authority  of  Israel,  has  been  developing  and 
supplying  high  technology  weapon  systems 
for  close  to  50  years.  With  a  combined  staff 
of  several  thousands  of  engineers,  scientists, 
technicians  and  other  skilled  personnel,  it  is 
the  largest  R&D  organization  in  Israel  for 
weapon  systems. 

At  RAFAEL  research  and  exploratory 
development  are  an  on-going  process. 
There  are  short  range  immediate  response 
projects  to  meet  the  urgent  needs  of  the 
Israeli  Defense  Forces  (IDF)  and  long  range 
projects  in  a  wide  range  of  disciplines. 
RAFAEL'S  products  and  systems  are  in 
operational  use  in  every  branch  of  the  IDF 
and  by  other  customers  both  in  Israel  and 
abroad.  The  weapon  systems  have  been 
proven  in  combat  time  and  time  again. 

Rafael  offers  a  wide  range  of  weapon 
systems  in  various  fields,  including  missiles, 
rockets,  EW,  special  purpose  computers 
and  thermal  imajging  systems.  Well  known 
products  include  the  PYTHON  4  air  to  air 
missile,  BARAK/ADAMS  vertically 


launched  point  defense  missile,  POPEYE  air 
i:o  ground  missile  and  more. 

The  most  modem  facilities  backup 
RAFAEL'S  technical  know-how.  World 
class  simulation  centers,  antenna 
(mgineering  facilities,  anechoic  chambers, 
iind  specialized  laboratories  dedicated  to  the 
development  of  electro-optics, 
micro-electronics,  computer  technologies, 
lire  control  and  missile  propulsion  and  the 
Weapon  Test  and  Evaluation  Center 
(WTEC),  the  National  over  land  test  range. 
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WTEC  Preseatation 


WTEC  is  located  in  the  southern  part  of 
Israel,  in  a  practically  uninhabited  desert 
area.  It  comprises  450  sq.  km  of  ground 
area,  of  various  terrain  textures  and 
landscapes,  from  which  more  than  10000 
sq.  km  of  air  space  are  controlled.  Large 
military  fire-zones  adjacent  to  the  test 
range  enable  extension  of  test  ground  area 
whenever  required.  The  test  range  is 
accessible  either  by  road  from  major  cities 
in  Israel,  or  by  air  using  one  of  two  nearby 


The  area  wiihin  the  range  is 
accessible  for  visual  inspection  and 
measurement  of  fragment  dispersion  of 
sophisticated  warheads. 

The  clear,  dry  desert  climate,  with  only  10 
days  of  rain  per  year,  offers  excellent 
visibility  and  allows  maximum  utilization  of 
the  high  precision  optical  instrumentation. 


The  main  test  range  facility  has  a  trilateral 
configuration  with  three  fixed  sites  located 
on  ridges  overlooking  the  central  impact 
zone  and  in  advimce  prepared  sites  which 
allow  deploying  the  mobile  instrumentation 


in  order  to  cover  optimally  the  planned 
flight  path. 

The  test  range  size  and  environment  enable 
the  operation  of  high-energy  missiles,  with 
minimal  outside  interference. 

Particularly  impressive  is  the  wide  range  of 
sophisticated  equipment,  both  fixed  and 
mobile,  which  together  provide  optimal 
geometry  tailored  for  each  test  mission. 
WTEC  operates  various  t5q)es  of  measuring 
instruments  and  data  gathering  devices,  as 
well  as  EW  threats  and  emitters. 

The  test  range  instrumentation  includes 
optical  tracking  mounts,  cinetheodolites, 
instrumentation  radars,  tactical  search 
radars,  telemetry  receiving  stations, 
meteorology  and  surveying  geodetic 
equipment. 

EW  threats,  fixed  and  mobile,  consisting  of 
replicas  as  well  as  authentic  systems,  allow 
system  survivability  testing  under  realistic 
threat  environment  simulation. 

A  large  arsenal  of  various  high  speed  film 
and  video  cameras,  shuttered  video  and  IR 
cameras  increase  electro-optical 

instrumentation  versatility  and  data 
accuracy,  and  shorten  the  analysis 
time-cycle. 

All  the  tracking  systems,  including  the  EW 
systems,  are  linked  and  controlled  by  the 
computerized  Command  and  Control 
Center.  The  command  rooms  contain 
consoles  with  color  graphic  displays,  video 
monitors,  real-time  telemetry  displays  and 
various  types  of  communication  means. 

The  core  of  the  control  computer  system, 
consists  of  two  VAX  4000  series 
computers,  which  together  with  double 
peripherals  provide  backup,  as  well  as 
ample  off-line  computing  power.  The  VAX 
computers  constantly  update  the  display 
stations  in  the  control  rooms,  by  data 
broadcast  over  a  local  area  network 
system. 

Mutual  slaving,  data  processing  in  real 
time,  graphic  and  alphanumeric  displays  of 
all  measurements  and  simultaneous  tracking 
of  multiple  targets,  all  in  real  time,  are 
among  the  many  capabilities  of  WTEC. 
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History  background. 


Testing  at  Shdema  test  site  began  in  the 
inid  1950's  with  only  tents  for  lodging  and 
jeeps  for  transportation. 

^iVTEC  began  as  a  small  photographic  unit 
wth  a  mission  to  test  simple  weapons 
which  were  developed  by  the  young  Israeli 
jirmament  industiy^  at  this  time. 

Small  guns,  rockets,  grenades  and  mines 
were  tested  and  evaluated  at  the  test  site. 

In  the  mid  70's  the  first  instrumentation 
radars  and  cinetheodolites  were  purchased 
and  integrated  iiiio  the  test  range. 

During  the  80's  all  tracking  instruments 
were  computerbied  and  connected  via 
communication  lines  to  a  central  computer. 
After  this  stage  the  method  of  mutual 
slaving  by  a  strong  central  computer  could 
be  easily  adopted  and  implemented. 

In  1988,  as  a  part  of  major  changes  in 
RAFAEL,  WTEC  was  declared  to  be  a 
profit  center.  How  this  decision  effected 
the  future  will  be  described  below. 


Inl991  the  mobile  test  range  had  its  first 
appearance.  It  was  during  the  Desert  Storm 
War,  that  the  mobile  test  range  was  moved 
to  the  Tel  Aviv  area  to  track  the  incoming 
SCUD  missiles  and  examine  the 
performance  of  the  Patriot  missiles  being 
launched  against  them.  This  was  a  real 
challenge  for  our  operators  to  perform  their 
mission  in  a  battle  situation. 

In  1994  the  Electronic  Warfare  Test  and 
Training  Facility  was  merged  with  WTEC  . 
This  was  the  drive  and  the  opportunity  to 
cjstablish  the  Integrated  Test  and  Training 
Range  (ITTR).  .  The  mission  of  ITTR  is  : 

-  Test  and  evaluation  of  weapons 
under  realistic  operational 
conditions,  including  real  threats 
and  replicas. 

-  Analysis  and  evaluation  of  EW 
systems 

-  Training  of  aircrews  and  EW 
operators 

-  Real  time  control  and  debriefing  for 
ground  army^  exercises. 
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Management  decisions. 

Presented  here  are  two  major  managerial 
decisions ,  which  were  taken  during  the  last 
decade.  These  decisions  have  changed  the 
management  infrastructure  of  the  facility, 
had  a  major  influence  on  the  test  range 
development  and  have  shaped  its  position 
as  the  leading  test  and  evaluation  facility  in 
Israel. 

The  first  decision  was  that  WTEC  become 
a  profit  center  and  act  as  an  independent 
economic  entity  (1989).  The  second 
decision  was  to  merge  the  EW  test  and 
ucuning  facility  with  WTTiC  (1994). 

WTEC  -  A  profit  center 

Over  the  past  decade  the  Israeli  Ministry  of 
Defense  and  RAFAEL  have  been  facing 
reduced  budgets,  resulting  in  a  decline  in 
the  numbers  of  systems  developed  and 
tests  performed  at  WTEC. 

Until  1987  WTEC  did  not  govern  itself 
according  to  economical  principles.  All  the 
services  were  givcm  to  MOD  organizations 
without  charge.  The  decision  that  WTEC 
will  be  a  profit  center  was  part  of  the 
larger  step  taken  by  IMOD  to  convert 
RAFAEL  status  from  a  department  in 
IMOD  to  a  frilly  owned  government 
company. 

This  decision  which  was  not  implemented 
as  of  now  caused  nevertheless  a  major 
change  and  had  an  extreme  influence  on 
WTEC. 

We  had  to  redefine  all  the  processes  and 
tasks  of  the  test  range  personnel,  yielding 
a  dramatic  reduction  of  personnel  (over 
40%)  which  included  recruiting  young  and 
skilled  engineers  £ind  technicians, 

We  outsourced  a  lot  of  functions  which 
were  not  part  of  our  core  competencies,  so 
we  could  focus  on  our  main  expertise.  For 
example,  we  decided  to  outsource  all  the 
activities  concerned  Avith  food,  lodging, 
garage,  depots,  transportation  (partially) 
etc. 


The  savings  from  this  activity  reached  over 
10  percent. 

IVe  changed  our  organizational  structure 
into  a  compact  and  efficient  one.  The 
organization  chart  became  simpler,  with 
less  distinct  groups  leading  to  higher 
operational  effectiveness  and  smaller 
overhead  cost. 

We  have  changed  the  organization  culture 
by  addressing  the  workforce  moral  and 
establishing  open  communications  and 
decision  sharing.  We  established 
commercial  charge  policy,  and  charged 
money  for  our  services.  We  initiated  new 
activities,  such  as  operational  testing  and 
monitoring  services  for  ground  forces 
training. 

We  have  fully  adopted  the  total  quality 
approach  in  the  test  and  evaluation 
activities,  with  an  emphasized  focus  on  the 
customer  and  his  needs.  We  undertook  a 
new  approach  towards  the  T&E  process  by 
defining  it  as  a  service  given  to  the 
customer. 

This  attitude  was  fiilly  adopted  and 
assimilated  at  all  levels  of  the  test  range 
personnel  ,  starting  with  the  management 
level  down  to  the  instrumentation  operators 
in  the  field. 

The  results  of  all  these  activities  were  very 
impressive. 

Within  three  years  the  test  range  became  a 
professional,  compact,  efficient  and 
effective  organization. 
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Merger  with  the  EW  facility. 

The  second  decision  was  the  merging  with 
the  EW  facility.  This  decision  was  taken 
when  WTEC  was  already  an  efficient 
organization.  The  merger  was  done  by 
including  the  EW  facility  as  a  part  of  the 
instrumentation  department  of  WTEC. 

This  had  two  major  impacts: 

-  The  merger  required  an  intensive  and 
comprehensive  development  effort,  which 
resulted  in  the  establishment  of  the 
Integrated  Test  and  Training  Range.  Most 
of  the  development  was  performed  by 
WTEC  personnel  and  this,  by  its  own 
virtue,  had  a  positive  impact  on  the  test 
range  activities. 

-  Due  to  the  similarity  between  the  two 
organizations  smd  the  increase  in 
productivity  (training  and  motivation)  we 
could  reduce  our  combined  personnel  by  an 
additional  15  percents  and  increase  the 
number  of  missions  that  we  were  able  to 
perform . 

This  last  decision  was  the  prime  cause  that 
led  WTEC  to  become  economically 
balanced.  Today  we  are  proud  to  have  a 
facility  which  is  one  of  its  kind  in  the 
world,  operated  by  a  small  but  very 
professional  and  Idghly  motivated  group. 


Summary 

WTEC  has  grown  together  with  weaponry 
development  in  Israel  and  became  and 
advanced  and  sophisticated  test  range 
which  enables  to  test  and  evaluate  the  most 
advanced  weapons  and  EW  systems. 

Two  major  managerial  decisions  have 
shaped  our  test  range,  its  activities  and 
performance. 

Acting  as  a  profit  center  led  the 
management  to  look  for  ways  and  means 
to  increase  productivity  and  to  reengineer 
the  T&E  process.  Managers  leadership  and 
personnel  responsibility  and  maturity 
caused  this  mission  to  be  successful. 

Now  we  are  in  the  process  of  defining  and 
designing  the  Virtual  Integrated  Test  and 
Training  Range  (VITTR)  which  will  enable 
missions  in  a  combined  real-virtual  domain. 
This  will  be  accomplished  by  connecting 
the  ITTR  with  Israeli  National  Battle 
Laboratory  (INBL)  and  additional 
simulator  facilities  in  Israel. 
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Abstract 

The  Joint  Advanced  Distributed  Simulation 
Joint  Test  Force  (JADS  JTF)  is  chartered  by  the 
U.S.  Office  of  the  Secretary  of  Defense  (OSD) 
to  determine  the  utility  of  advanced  distributed 
simulation  (ADS)  for  both  developmental  and 
operational  test  and  evaluation  (DT&E  and 
OT&E).  The  program  is  at  its  midpoint,  and 
this  paper  is  designed  to  provide  a  progress 
report  on  the  lessons  learned  to  date  on  the  use 
of  ADS  in  test  and  evaluation  (T&E). 

The  paper  opens  with  a  brief  overview  of  ADS 
technology  and  then  a  short  description  of  the 
JADS  Joint  Test  and  Evaluation  (JT&E) 
program.  Third,  the  main  portion  of  the  paper 
will  discuss  the  results  and  lessons  learned 
during  the  ADS-enhanced  testing  conducted 
throughout  the  first  major  phases  of  the  JADS 
JT&E  program.  Fourth,  the  JADS  study  on  the 
linking  of  electronic  warfare  (EW)  test 
facilities,  the  Threat  Systems  Linking 
Architecture  (TSLA)  Study,  is  briefly 
described.  Finally,  other  considerations  will  be 
offered  for  the  T&E  professional  interested  in 
whether  ADS  might  be  a  suitable  test  tool. 

The  material  in  this  paper  fuses  material  from 
other  JADS  documents  prepared  by  many 
members  of  the  JADS  JTF.  Readers  are 
encouraged  seek  more  information  JADS  has 
compiled  either  via  the  address  above  or  their 
web  site  (http://www.jads.abq.com). 

Overview  of  ADS  (Ref.  1) 

Since  the  mid- 1980s,  rapidly  evolving 
information  systems  technology  has  been  put  to 
work  in  support  of  U.S.  Department  of  Defense 


(DoD)  needs.  Early  efforts  were  conducted 
jointly  by  the  Defense  Advanced  Research 
Projects  Agency  and  the  U.S.  Army.  This  early 
project  was  named  Simulation  Network 
(SIMNET),  and  it  was  sharply  focused  on 
training  applications.  Conceptually,  the  project 
was  directed  toward  linking  training  devices 
(simulators)  with  human  operators  in  the  loop  at 
distributed  sites  in  a  common  virtual 
environment  in  near  real  time.  SIMNET 
evolved  to  distributed  interactive  simulation 
(DIS),  a  technology  implementation  which  is 
more  flexible  and  far  reaching.  Formal  industry 
standards  have  been  established  for  DIS.  In 
turn,  DIS  is  giving  way  to  high  level 
architecture  (HLA),  a  technical  approach 
championed  by  the  U.S.  Defense  Modeling  and 
Simulation  Office. 

JADS  uses  a  more  generic  term  for  the 
technology  -  ADS.  This  is  defined  as  the 
technology  and  procedures  that  provide  a  time 
and  space  coherent,  interactive  synthetic 
environment  through  geographically  distributed 
and  potentially  dissimilar  situations.  Any 
combination  of  live,  virtual,  or  constructive 
simulation  of  people  and/or  equipment  can  be 
used.  ADS  is  the  concept;  DIS  and  HLA  are 
applications  of  ADS. 

Overview  of  JADS  JT&E 
Background  (Ref.  1) 

Because  of  widespread  interest  in  using  ADS 
technology  to  support  T&E,  the  JADS  JT&E 
program  was  nominated  for  feasibility  study  in 
1993.  The  nomination  was  motivated  by  the 
T&E  community's  concern  about  long-standing 
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test  constraints  and  limitations,  and  the  potential 
utility  of  ADS  for  relieving  some  of  those 
constraints  and  limitations.  However,  there  was 
widespread  skepticism  that  ADS  might  not  be 
able  to  deliver  high-quality  data  demanded  by 
the  T&E  community.  The  Services  concurred 
with  the  need  for  a  rigorous  examination  of 
ADS  utility  to  testing,  and  OSD’s  Director  of 
Test,  System  Engineering  and  Evaluation 
chartered  JADS  as  a  full  joint  test  program. 

JAPS  JT&E  Charter  (Ref.  2) 

The  basic  JADS  JT&E  program  was  chartered 
in  October  1994  to  investigate  the  utility  of 
ADS  for  both  DT&E  and  OT&E.  More 
specifically,  JADS  is  to  investigate  the  present 
utility  of  ADS,  to  identify  critical  constraints  in 
using  the  technology,  to  develop  the 
methodologies  in  using  ADS  in  various  T&E 
applications,  and  to  provide  growth 
requirements  for  ADS  so  that  as  it  matures  it 
better  meets  the  needs  of  the  T&E  community. 

At  the  time  of  chartering,  OSD  tasked  JADS  to 
investigate  the  possibility  of  specifically 
examining  ADS  utility  to  EW  T&E.  This 
additional  facet  of  the  program  was 
subsequently  chartered  in  August  1996  (Ref.  3). 

Test  Approach 

To  accomplish  this  charter,  JADS  is  conducting 
three  series  of  ADS-enhanced  tests  in  widely 
different  areas  to  determine  the  utility  of  ADS. 
Representative  “systems  under  test”  are  used, 
ones  that  have  already  undergone  testing  and 
have  been  fielded.  Significant  system 
performance  data  is  available  then  for 
comparison  with  the  data  obtained  in  the  tests 
introducing  ADS  as  a  methodology.  The  three 
specific  test  programs  are  the  System 
Integration  Test  (SIT)  utilizing  two  air-to-air 


missiles  (AIM-9M  Sidewinder  and  AIM-120 
Advanced  Medium  Range  Air-to-Air  Missile 
(AMRAAM));  the  End-to-End  Test  (ETE) 
using  the  Joint  Surveillance  Target  Attack 
Radar  System  (Joint  STARS)  as  a 
representative  command,  control, 

communications,  computer,  intelligence, 

surveillance  and  reconnaissance  (C4ISR) 

system;  and  the  Electronic  Warfare  (EW)  Test, 
utilizing  the  ALQ-131  self-protection  jammer 
(SPJ). 

System  Integration  Test  (Ref.  4) 

Sir  evaluated  the  utility  of  using  ADS  to 
support  cost-effective  testing  of  an  integrated 
missile  weapon/launch  aircraft  system  in  an 
operationally  realistic  scenario.  The  purpose  of 
Sir  also  included  the  evaluation  of  the 
capability  of  the  JADS  Test  Control  and 
Analysis  Center  (TCAC)  to  control  a  distributed 
test  of  this  type  and  to  remotely  monitor  and 
analyze  test  results. 

Sir  consisted  of  two  phases,  each  of  which 
culminated  in  fully  linked  missions.  The 
missions  simulated  a  single  shooter  aircraft 
launching  an  air-to-air  missile  against  a  single 
target  aircraft.  In  the  Linked  Simulators  Phase 
(LSP),  the  shooter,  target,  and  missile  were  all 
represented  by  hardware-in-the-loop  (HWIL) 
laboratories.  LSP  testing  was  completed  in 
November  1996.  In  the  Live  Fly  Phase  (LFP), 
the  shooter  and  target  were  represented  by  live 
aircraft  and  the  missile  by  a  HWIL  laboratory. 
LFP  testing  was  completed  in  October  1997. 

Linked  Simulators  Phase  The  LSP  test  concept 
was  to  replicate  a  previous  AIM-9M-8/9  live 
fire  profile  in  an  ADS  configuration  and 
compare  missile  results  for  the  LSP  trials  to 
those  from  the  live  fire  test.  The  LSP  test 
configuration  is  shown  in  Figure  1. 
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Figure  1.  Linked  Simulator  Phase  Configuration 


The  F/A-18  Weapon  System  Support  Facility 
(WSSF)  at  China  Lake,  California,  and  the  F- 
14D  Weapon  System  Integration  Center  (WSIC) 
at  Point  Mugu,  California,  were  the  shooter  and 
target,  respectively.  The  shooter  “fired”  the 
AIM-9  in  the  Simulation  Laboratory  (SIMLAB) 
HWIL  facility  at  the  target  which  could  respond 
with  countermeasures.  Runs  were  controlled 
from  a  test  control  center  which  ensured  all 
nodes  were  ready  for  each  run,  issued  start/stop 
directions,  and  processed  data  packets  for  real¬ 
time  analysis  of  system  performance.  Test 
control  was  exercised  from  the  Battle 
Management  Interoperability  Center  (BMIC)  at 
Point  Mugu  while  the  JADS  Joint  Test  Force 
was  physically  relocating.  Control  switched  to 
the  JADS  TCAC  in  Albuquerque,  New  Mexico 
after  the  move  was  complete. 


Live  Fly  Phase  The  LFP  test  concept  was  to 
replicate  previous  AMRAAM  live  fire  profiles 
in  an  ADS  configuration  and  compare  missile 
results  from  the  LFP  trials  to  those  from  the  live 
fire  tests.  In  the  LFP,  ADS  techniques  were 
used  to  link  two  live  F-16  aircraft  (flying  on  the 
Gulf  Test  Range  at  Eglin  Air  Force  Base, 
Florida)  representing  the  shooter  and  target  to 
an  AMRAAM  HWIL  laboratory  (also  at  Eglin) 
representing  the  missile.  This  configuration 
allowed  data  from  live  sources  to  drive  the 
HWIL  laboratory  for  more  realistic  missile 
results  and  is  shown  in  Figure  2. 
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Figure  2.  Live  Fly  Phase  Configuration 

Global  positioning  system  (GPS)  and  telemetry 
(TM)  data  were  downlinked  from  the  aircraft 
and  passed  to  the  Central  Control  Facility 
(CCF)  at  Eglin.  GPS,  inertial  navigation  system 
(INS),  and  tracking  radar  data  for  each  aircraft 
were  combined  by  the  TSPI  (time-space- 
position  information)  Data  Processor  (TDP)  in 
the  CCF  to  produce  optimal  entity  state 
solutions.  The  aircraft  entity  state  data  were 
transformed  into  DIS  protocol  data  units 
(PDUs)  and  transferred  to  the  AMRAAM 
HWIL  simulation  at  the  Missile  Simulation 
Laboratory  (MISILAB)  over  a  T3  link. 

The  shooter  aircraft  “fired”  the  AMRAAM  in 
the  MISILAB  at  the  target  and  provided  data 
link  updates  of  the  target  position  and  velocity 
to  the  missile  during  its  flyout.  The  AMRAAM 
seeker  was  mounted  on  a  flight  table  and 
responded  to  radio  frequency  (RF)  sources  in 
the  MISILAB  which  simulated  the  seeker  return 
from  the  target,  the  relative  motions  of  the 


target  and  the  missile,  and  electronic 
countermeasures  (ECM).  A  link  between  the 
CCF  and  the  JADS  TCAC  allowed  IADS 
personnel  to  monitor  and  record  the  simulated 
intercepts. 

End-to-End  Test 

The  ETE  uses  distributed  simulations  to 
assemble  an  enhanced  environment  to  be  used 
for  testing  command,  control,  communications 
and  computer  (C4I)  systems.  The  object  is  to 
determine  if  ADS  can  provide  a  complete, 
robust  set  of  interfaces  from  sensor  to  weapon 
system  including  the  additional  intermediate 
nodes  that  would  be  found  in  a  tactical 
engagement.  The  test  traces  a  thread  of  the 
battlefield  process  from  target  detection  to 
target  assignment  and  engagement  at  corps  level 
using  ADS.  Figure  3  illustrates  the  basic  test 
architecture. 
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Figure  3.  End-to-End  Test  Architecture 

The  ETE  is  a  four-phased  test.  Phase  1  was 
largely  developmental  -  constructing  the 
various  components  necessary  to  executing  later 
phases  of  testing.  These  components  include  a 
high  fidelity  emulation  of  the  Joint  STARS 
radar  processes,  called  Virtual  Surveillance 
Target  Attack  Radar  System  (VSTARS),  which 
includes  both  moving  target  indicator  and 
synthetic  aperture  radar  modes  of  operation. 
Phase  2  links  representative  entities  for  the  end- 
to-end  process  while  the  “system  under  test”  is 
in  a  laboratory  environment  enabling  JADS  to 
explore  the  utility  of  ADS  in  the  DT&E  and 
early  OT&E  of  a  C4I  system.  Phase  3  hosts 
VSTARS  on  board  the  actual  Joint  STARS 
aircraft  and  performs  final  integration  testing. 
Phase  4  is  an  actual  live  open  air  test  with  the 
aircraft  airborne,  with  the  environment 
augmented  by  ADS. 

Electronic  Warfare  Test  (Ref.  6) 

JADS  EW  Test  was  chartered  separately  by 
OSD  to  examine  the  utility  of  ADS  in  EW 
T&E.  To  allow  JADS  to  conduct  a  broad 
analysis  of  this  domain  and  remain  within  very 
tight  fiscal  constraints,  a  “multi-vectored” 
approach  is  employed.  JADS  leveraged  off  the 


U.S.  DoD  High  Level  Architecture  (HLA) 
Engineering  Prototype  Federation  for  lessons 
learned  in  constructing  and  implementing  a 
distributed  architecture  for  EW  T&E.  At  the 
bequest  of  DoD’s  CROSSBOW  Committee, 
JADS  directed  the  TSLA  Study,  which 
delineates  how  to  link  DoD’s  EW  test  facilities 
using  the  HLA.  Third,  JADS  is  participating 
with  the  U.S.  Army  in  its  Advanced  Distributed 
Electronic  Warfare  System  (ADEWS)  test,  a 
concept  that  provides  EW  effects  on 
communications  gear  in  the  open  air 
environment  without  the  actual  EW  open  air 
emissions.  Fourth,  JADS  offers  test  agencies 
and  program  offices  “comparison  studies,” 
where  a  traditional  test  of  a  system  is  compared 
with  an  ADS-enhanced  test  to  identify  potential 
benefits  in  test  thoroughness,  time  and  money. 
JADS  Flag  Officer  Steering  Committee  directed 
that  these  studies  be  performed  after  the  JTF  has 
performed  its  self-protection  jammer  (SPJ)  test. 

SPJ  Test  (Ref.  7) 

The  SPJ  test  has  been  designed  as  a  three- 
phased  test  focusing  on  the  U.S.  DoD  EW  test 
process,  and  utilizes  the  ALQ-131  as  its  “system 
under  test.”  Phase  1  is  a  non-ADS  test  of  the 
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SPJ  on  an  open  air  range  (OAR),  augmented 
with  data  obtained  by  testing  the  ALQ-131  in  a 
hardware-in-the-loop  facility.  The  purpose  of 
this  test  is  to  establish  a  baseline  of  environment 
and  performance  data  which  will  be  used  to 
develop  the  ADS  test  environment  for  the 
following  phases  and  will  be  the  basis  for 
determining  the  validity  of  ADS  test  results. 
Phase  2  is  a  test  of  a  high-fidelity  real-time 
digital  system  model  (DSM)  of  the  ALQ-131 
linked  with  hardware-in-the-loop  terminal 
threats  and  a  constructive  model  of  an 
Integrated  Air  Defense  System  (IADS).  The 
threat  laydown  from  the  OAR  is  replicated  in 
the  synthetic  ADS  environment  and  the  ALQ- 


131  will  be  flown,  via  a  scripted  flight  profile 
developed  from  the  actual  OAR  flights,  through 
the  IADS,  engaging  the  high-fidelity  terminal 
threats.  Phase  3  is  a  test  of  the  SPJ  installed  on 
an  actual  aircraft  located  in  an  Integrated 
System  Test  Facility  (ISTF).  The  facility  will 
be  linked  with  hardware-in-the-loop  threats  and 
the  constructive  model  of  the  IADS  using  the 
same  threat  laydown  as  the  previous  test  and 
controlled  by  the  same  scripted  flight  profile. 
Figure  4  illustrates  Phases  2  and  3  of  the  SPJ 
test. 


Figure  4.  Self-Protection  Jammer  Phases  2  and  3 


JAPS  JT&E  Test  Results  As  the  first  phase  of  ETE  was  largely 

developmental,  this  section  will  focus  on  SIT 
At  the  time  of  this  writing,  JADS  has  completed  results.  (For  a  schedule  of  JADS  test  execution, 
one  phase  of  the  ETE  and  both  phases  of  SIT.  refer  to  Figure  5). 
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Figure  5.  JADS  Test  Execution  Schedule 


Linked  Simulators  Phase  Results  (Ref.  8) 

The  key  results  from  LSP  testing  were  as 
follows: 

-  The  simulation  facilities  were  properly 
linked,  and  the  missile  flyouts  were 
valid  for  the  target  representation  in  the 
Simulation  Laboratory  (SIMLAB). 
However,  this  target  representation 
differed  somewhat  from  the  target  data 
originating  from  the  Weapon  System 
Integration  Center  (WSIC). 

-  The  manual  method  for  replicating  a 
given  profile  resulted  in  very  good  run- 
to-run  reproducibility  of  the 
engagements. 

-  The  average  latency  of  all  entity  state 
data  during  the  final  mission  were 
relatively  small  (<100  milliseconds  from 
simulation  to  simulation)  and  consistent 
run-to-run.  However,  relatively  large 
random  latency  variations  were  often 
observed  which  resulted  in  an 
uncertainty  in  the  target  location,  as 
perceived  in  the  SIMLAB. 

-  The  ADS  network  provided  ample 
bandwidth  and  no  loss  of  connectivity 
during  testing. 


-  There  were  no  significant  ADS -induced 
errors. 

-  The  reliability  of  the  long-haul  network 
was  very  good,  and  the  availability  of 
the  complete  LSP  ADS  configuration 
was  on  the  order  of  85%. 

-  Test  control  procedures  were  refined 
throughout  the  preparation  process  and 
worked  well  during  testing. 

Live  Fly  Phase  Results  (Ref.  9) 

The  key  results  from  LFP  testing  were  as 
follows: 

-  The  live  aircraft  were  properly  linked  to 
the  missile  HWIL  laboratory,  and  the 
Missile  Laboratory  (MISILAB) 
generated  valid  AMRAAM  data  during 
the  engagement. 

-  Accurate  time-space-position 

information  (TSPI)  solutions  were 
generated  by  the  TSPI  Data  Processor 
(TDP)  to  the  order  of  one  to  three  meters 
in  position  and  one  meter  per  second  in 
velocity.  This  well  exceeded  MISILAB 
accuracy  requirements. 
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-  The  shooter  and  target  TSPI  data  were 
properly  synchronized  to  each  other  and 
to  the  umbilical  and  data  link  messages 
for  input  to  the  MISILAB  simulation. 

-  Latencies  during  testing  were  relatively 
stable  and  consistent,  but  fairly  large. 
The  total  latency  of  the  MISILAB 
simulation  was  about  3.1  seconds.  This 
large  value  of  latency  was  due  to  the 
processing  and  buffering  of  the  TSPI 
data  to  produce  accurate  and  smooth 
solutions  and  to  the  synchronization 
technique  used. 

-  The  ADS  network  provided  ample 
bandwidth  and  no  loss  of  connectivity 
during  testing. 

-  There  were  no  significant  ADS-induced 
errors. 

-  Test  control  procedures  worked  well 
during  testing  with  centralized  test 
control  exercised  from  the  CCF. 


System  Integration  Test  Lessons  Learned 
LSP  Lessons  Learned 

LSP  lessons  learned  are  documented  in  much 
detail  in  JADS  LSP  Final  Report  (Ref.  10)  and 
are  categorized  in  the  general  domains  of 
technical  and  infrastructure  lessons  learned. 
What  follows  are  some  of  the  highlights. 

LSP  Technical  Lessons  Learned 

-  Accurate  coordinate  transformations  are 
necessary.  They  must  be  verified  and  validated 
at  each  site  and  then  revalidated  during  end-to- 
end  testing  as  early  as  possible  in  the  test  phase. 

-  Quantitative  validation  has  limitations. 
JADS  intent  was  to  quantitatively  verify  missile 
simulation  performance  against  live  fire  data. 
However,  as  only  one  live  fly  event  was 
available  to  support  the  process,  a  modified 
approach  including  both  quantitative  and 
qualitative  methods  was  used,  and  successfully 
identified  invalid  results. 

-  Network  interface  units  (NIUs)  need 
improvement.  NIUs  are  necessary  if  two  nodes 


cannot  communicate  directly  in  a  common 
language.  They  can  be  a  major  source  of  both 
errors  and  processing  delays.  Better  direct  user 
control  of  the  content  of  the  data  and  network 
communications  is  needed. 

-  Common  ADS -related  hardware  and 
software  is  needed.  In  the  LSP,  it  was  difficult 
to  get  the  ADS  network  to  behave  in  a  uniform 
fashion  due  to  the  many  different  types  of 
interface  hardware,  communications  equipment 
(routers),  and  interface  software  versions. 

-  Latency  variations  were  significant. 
Processing  delays  were  the  primary  culprit  here. 

-  Time  sources  must  be  synchronized  off  the 
same  time  source  and  then  must  be  validated  at 
each  test  site  prior  to  project  operations  to 
ensure  accurate,  synchronized  time  is  precisely 
recorded  at  each  test  site. 

-  Special  test  equipment  is  needed  for  check¬ 
out  and  verification  of  the  ADS  architecture. 
Without  this  equipment,  trial  and  error  becomes 
the  norm  when  (not  if)  problems  crop  up. 

LSP  Infrastructure  Lessons  Learned 

-  The  requirements  for  an  ADS  test  must  be 
clearly  defined  early  in  the  test  planning  phase. 
This  includes  user  requirements,  support 
agency’s  stated  actions,  and  operations  security 
requirements.  Planning  and  coordination  details 
will  be  much  more  involved  than  in  a 
traditional,  non-ADS  test. 

-  Get  “system  under  test”  experts  involved 
from  the  beginning. 

-  Test  communications  requirements  must  be 
addressed  early  in  the  test  planning  phase.  This 
is  necessary  to  ensure  effective  communications 
during  the  test.  Also,  a  linked  test  should  have 
multiple  (more  than  two)  communications  nets 
with  easy,  selectable  access  to  all  the  nets  from 
multiple  locations  within  the  site.  Finally,  the 
capability  for  secure  video  teleconferencing 
pays  big  dividends  during  planning, 
coordination,  and  post-test  debriefs. 

-  A  stepped  build-up  approach  should  be 
used.  First,  a  systematic  check-out  of  the  stand¬ 
alone  simulators  (live,  virtual  or  constructive)  is 
needed.  Next,  direct  (non-DIS)  links  should  be 
used  during  test  build-up.  Finally,  structured 
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testing  of  the  network  must  be  performed  prior 
to,  and  independent  of,  the  linked  testing  times 
and  the  simulation  laboratories  to  validate 
transmission/reception  rates,  bandwidth 
utilization,  latency,  data  transmission  and 
reception,  etc.,  prior  to  commencing  project  test 
periods. 

-  Linking  of  facilities  using  ADS  can  require 
significant  facility  interface  hardware  and 
software  development.  ADS  implementation  is 
not  “plug  and  play,”  at  least  for  some  time. 

-  Local  (on-site)  test  monitoring/control 
should  be  used  prior  to  remote  test 
monitoring/control. 

-  Tight  control  of  the  aircrew  is  not  desirable. 
Give  them  the  critical  parameters  and 
switchology  to  meet  the  test  objectives  and 
allow  them  to  make  tactical  decisions,  fly  the 
“aircraft,”  operate  the  weapon  system,  etc. 

-  Additional  time  is  needed  before  the 
beginning  and  after  the  end  of  each  testing 
period.  One  hour  is  recommended  for  set-up, 
and  two  hours  at  the  end  for  data  logging,  data 
archiving,  data  transfer,  and  laboratory 
reclassification. 

-  Briefings  are  needed  before  and  after  each 
mission. 

-  Effective  data  management  is  needed,  as 
ADS  can  generate  mountains  of  data.  A 
comprehensive  plan  will  clearly  identify  the 
data  to  be  collected  at  each  site,  on-site 
processing  of  the  data,  and  data  to  be  transferred 
to  the  analysis  center. 

-  Adequate  time  must  be  allowed  for  data 
analysis  between  test  events.  Analysis 
procedures  should  be  rehearsed  to  better 
understand  the  amount  of  time  needed  for  this 
analysis. 

-  Configuration  control  is  essential.  This  one 
obvious  area  was  one  of  great  challenge 
considering  the  many  sites  involved  and  the 
multiple  uses  of  each  site. 

LFP  Lessons  Learned 

LFP  lessons  learned  are  documented  in  much 
detail  in  JADS  LSP  Final  Report  (Ref.  1 1)  and 
are  also  categorized  in  the  general  domains  of 


technical  and  infrastructure  lessons  learned. 
Some  of  the  highlights  are  listed  below. 

LFP  Technical  Lessons  Learned 

-  As  in  the  LSP,  a  major  lesson  learned  is  that 
stand-alone  simulation  facilities  (for  live,  virtual 
or  constructive  entities)  can  require  significant 
modifications  before  effective  linking  is 
possible. 

-  Additionally,  linking  may  require  special 
purpose  interfaces  so  as  to  accept  inputs  in  real 
time.  Development  of  such  units  must  be 
factored  into  test  planning. 

-  Key  interfaces  need  realistic  integration 
testing.  Replaying  data  from  a  recorded  mission 
worked  well  in  most  cases  (and  was  most  cost 
effective);  however,  some  integration  testing 
required  a  live  mission. 

-  Early  definition  of  network  requirements 
was  very  advantageous.  This  was  a  major 
lesson  from  LSP  that  JADS  took  advantage  of. 

LFP  Instrumentation  Lessons  Learned 

Changes  and  upgrades  to  aircraft 
instrumentation  delayed  development. 
Specially  instrumented  aircraft  were  required  to 
support  the  LFP  flights.  Due  to  the  small 
number  of  such  aircraft,  the  LFP  schedule  was 
very  sensitive  to  periodic  aircraft  phase 
inspections,  software  upgrades,  and  higher 
priority  missions. 

-  Merging  several  TSPI  sources  was 
advantageous.  Real-time  aircraft  inertial 
navigation  system  (INS)  and  global  positioning 
satellite  (GPS)  data  were  combined  to  calculate 
more  accurate  kinematic  estimates.  When 
combined  with  the  ground  radars,  solutions  of 
one  to  three  meters  in  position  and  one  meter 
per  second  in  velocity  were  achieved. 

-  A  strong  program  manager  or  system 
integrator  is  needed  to  oversee  facility 
development,  due  to  the  difficulty  in 
coordinating  several  diverse  facilities  to 
successfully  integrate  an  ADS -linked 
configuration. 

-  Use  risk  reduction  tests  for  integration.  A 
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building  block  approach  was  used  successfully 
to  check  out  interfaces  at  the  lowest  level,  then 
one  or  two  resources  at  a  time  were  added  to 
integrate  the  linked  configuration.  These  risk 
reduction  tests  were  also  useful  for  developing 
analytical  tools. 

-  Several  subnetworks  should  be  used  for 
voice  communications.  Three  voice 
communications  networks  were  needed  to 
support  more  than  30  people  at  various 
locations,  and  a  fourth  network  could  have 
further  aided  decision  making. 

-  Two-dimensional  displays  were  needed  at 
each  node;  they  greatly  enhanced  the  situational 
awareness  of  the  participants. 

-  Existing  range  procedures  had  to  be 
modified  for  ADS.  The  existing  test  procedures 
were  only  written  for  individual  facilities,  so  a 
new  combined  checklist  was  created  for  ADS 
applications. 

-  Laboratory  replays  served  as  an  excellent 
method  of  test  rehearsal. 

Other  Topics  for  Consideration 

Threat  Simulator  Linking  Activities  CTSLA^ 
Study  (Ref.  12) 

TSLA  is  a  study  chartered  by  the  U.S.  DoD’s 
CROSSBOW  Committee  and  directed  by  the 
JADS  JTF.  The  TSLA  study  provides  an  ADS 
Capabilities  Assessment  Report  which  describes 
the  utility  of  ADS  in  the  context  of  the 
evolutionary  acquisition  process  interwoven 
with  T&E.  At  each  phase  of  the  acquisition 
process,  the  conventional  and  ADS 
methodologies  are  applied.  For  each  phase,  the 
necessary  test  facilities  are  delineated  and  the 
differences  in  test  capabilities  noted.  Test 
facility  requirements  are  addressed  for  SPJs, 
stand-off  jarmners,  and  integrated  avionics.  As 
the  test  facility  requirements  are  reviewed,  any 
improvements  needed  to  meet  the  requirements 
of  the  electronic  combat  test  process  are  noted. 
Some  of  these  improvements  are  needed 
without  regard  for  ADS.  In  other  cases,  the 
improvements  are  needed  only  to  support  ADS. 
Comparisons  of  capabilities,  with  and  without 


ADS,  are  discussed.  General  assessments  of  the 
cost  impact  of  ADS  are  also  discussed. 
Requirements  for  and  the  impact  of  latency  are 
also  discussed.  Latency  will  be  present. 
Depending  on  the  network  topology,  physical 
communication  infrastructure,  and  network 
management  methods,  it  may  be  possible  to 
achieve  a  tolerable  latency  for  most 
applications.  Latency  remains  the  greatest 
technological  risk  to  the  successful  use  of  ADS 
in  EW  T&E. 

Considerations  for  the  T&E  Professional 

Although  JADS  still  has  another  year  of  ADS 
testing  ahead  of  it,  there  are  several  overall 
considerations  which  have  become  evident. 
What  follows  are  some  highlights,  as  recently 
briefed  to  JADS  Flag  Officer  Steering 
Committee  (Ref.  13),  and  adopted  into  a 
pamphlet  entitled  “Emerging  Findings  from 
JADS.” 

-  ADS  allows  one  to  link  live,  virtual,  and 
constructive  players  based  on  need.  ADS  does 
not  mean  linking  together  several  constructive 
simulations  to  make  a  bigger,  more  complex 
model.  Rather,  it  means  blending  live,  virtual 
and  constmctive  players  to  give  the  user  the 
right  mix  of  fidelity  and  realism  to  meet  specific 
needs. 

-  Distribution  is  not  a  function  of  distance. 
Latency  is  a  function  of  processing  and 
transmission,  and  processing  latency  dominates. 
Transmission  latencies  are  predictable  and 
relatively  well  behaved.  Processing  latencies 
can  be  problematic,  though,  and  require  a 
thorough  understanding  of  the  individual  sites 
and  the  ADS  architecture.  This  holds  tme 
whether  the  network  covers  a  continent  or 
multiple  nodes  at  a  single  location. 

-  Validating  against  live  data  is  problematic. 
Problems  include  the  quality  of  the  live  data  and 
the  lack  of  data  availability. 

-  Data  collection  is  different  from  traditional 
T&E  and  training.  Generally  speaking,  an  ADS 
environment  is  easier  to  instrument  than  the 
traditional  live  test  environment  and  provides 
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more  trials  per  unit  of  time.  The  end  result  is 
that  analysts  can  get  inundated  with  data.  In 
addition,  ADS  testing  requires  additional  data  to 
be  collected  on  the  performance  of  the  networks 
linking  your  sites. 

-  ADS  cost  benefits  are  best  realized  over  the 
entire  system  life  cycle.  JADS  has  performed 
some  cost  benefit  analyses  comparing 
traditional  test  approaches  to  ones  utilizing 
ADS.  In  many  cases,  it  appears  as  if  ADS  could 
save  time  and  money  (as  well  as  allowing  a 
more  rigorous  test)  in  just  the  test  phase  of  a 
development  program.  However,  the  full 
benefits  of  using  ADS  would  be  realized  over 
all  the  phases  of  the  acquisition  cycle,  from 
requirements  development  to  training  and 
sustainment.  This  supports  the  concepts 
advocated  in  both  simulation  based  acquisition 
(SBA)  and  Simulation,  Testing  and  Evaluation 
Program  (STEP). 

-  ADS  allows  one  to  test  “differently.” 
Adding  ADS  to  a  traditional  test  approach 
provides  only  a  fraction  of  the  value  ADS  can 
bring  to  bear.  To  realize  the  full  capabilities  of 
this  enabling  technology,  one  will  construct  a 
test  event  fundamentally  differently  than  its 
traditional  forefather. 

-  To  a  certain  extent,  latency  is  manageable. 
The  ADS  architectural  design  is  the  most 
determining  factor  of  latency.  The  tester  must 
approach  network  design  from  a  requirements 
viewpoint.  Based  on  what  the  tester  is  trying  to 
accomplish,  an  architecture  can  usually  be 
designed  which  balances  the  types  of 
participating  assets,  fidelity  requirements,  and 
tolerable  latency. 

-  The  effect  of  latency  is  dependent  on  the 
players  involved.  Latency  is  a  factor  when  the 
tester  is  trying  to  generate  closed-loop 
interactions.  Again,  the  tester  must  approach 
the  test  from  a  requirements  viewpoint  in 
determining  whether  an  ADS  architecture  can 
provide  the  interactions  needed  for  the  test 
event. 


JADS  has  been  chartered  to  determine  the  truth 
as  to  where  ADS  is,  or  is  not,  a  feasible  tool  for 
T&E.  JADS  JT&E  testing  is  well  underway, 
and  the  early  evidence  is  that  ADS  can  bring 
many  benefits  to  the  table.  However,  one  must 
be  fully  aware  of  the  inherent  limitations  of  the 
technology.  Also,  one  would  be  well  advised  to 
learn  from  those  who  have  practical  experience 
in  using  ADS  in  the  T&E  arena.  Distributed 
simulation  is  certainly  not  a  panacea  that  will 
solve  all  of  the  problems  and  meet  all  of  the 
requirements.  However,  it  does  appear  to  be  a 
powerful  tool,  if  used  appropriately  and 
intelligently,  and  should  be  considered  in 
balance  with  other  methodologies  when 
developing  a  T&E  program. 
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Following  are  acronyms  found  in  this  paper’s 
figures  and  not  defined  in  the  body  of  the  paper. 

Figure  1 

A/C  -  Aircraft 
IR  -  Infrared 
MSL  -  Missile 

SMS  -  Stores  Management  System 
TGT  -  Target 

Figure  2 

AASI  -  Aircraft  Avionics  Simulation  Interface 
ECM  -  Electronic  Countermeasures 
RDL  -  Rear  Data  Link 
UMB  -  Umbilical 
TM  -  Telemetry 

Figure  3 

ACE  -  Analysis  and  Control  Element 
ASAS  -  All-Source  Analysis  System 
ATACMS  -  Army  Tactical  Missile  System 
DOCC  -  Deep  Operation  Coordination  Center 
FDC  -  Fire  Direction  System 
FSE  -  Fire  Support  Element 
GSM  -  Ground  Station  Module 
LOS  -  Line  of  Sight 
OK  -  Oklahoma 

SATCOM  -  Satellite  Communications 
SCDL  -  Surveillance  Control  Data  Link 
TRAC  -  U.S.  Army  Training  and  Doctrine 
Command  Analysis  Center 
WSMR  -  White  Sands  Missile  Range 

Figure  4 

ECM  -  Electronic  Countermeasures 
IADS  -  Integrated  Air  Defense  System 
RF  -  Radio  Frequency 
TECH  -  Technique 


Figure  5 
FY  -  Fiscal  Year 
Qtr  -  Quarter 
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ABSTRACT 

The  US  DoD  has  developed  a  GPS-based  TSPI 
system  for  use  on  major  T&E  DoD  ranges.  BAe 
Flight  Test  Department  is  purchasing  a  similar 
system.  BAe  desired  to  have  a  demonstration  of  the 
DoD  system.  The  two  organizations  agreed  to 
conduct  this  demonstration  in  the  February  and 
March  1996  time  period.  The  demonstration  took 
place  at  the  British  Aerospace  Military  Aircraft 
Division,  Warton  Flight  Test  Center,  Lancashire, 
Engiand.  This  paper  describes  the  assets  used  for 
the  demonstration  and  the  results  of  the 
demonstration. 

INTRODUCTION 

The  British  Ministry  of  Defense  (MoD)  and  the  US 
Department  of  Defense  (DoD)  established  a  Foreign 
Military  Sales  (FMS)  case  for  the  purpose  of 
demonstrating  a  DoD  GPS-based  instrumentation 
system  in  the  United  Kingdom  (UK).  The  tri-service 
Range  Applications  Joint  Program  Office  (RAJPO) 
is  responsible  for  developing  and  delivering  GPS- 
based  Time,  Space,  Positioning  Information  (TSPI) 
assets  for  use  on  the  major  DoD  test  and  evaluation 
(T&E)  ranges.  The  RAJPO  was  responsible  for 
conducting  the  demonstration. 

This  instrumentation  is  for  use  on  high,  medium  and 
low  dynamic  aircraft,  ships,  and  land  vehicles. 
Reference  A  describes  the  details  of  the  system 
and  provides  a  description  of  the  accuracy 
validation  tests  that  took  place  at  Eglin  Air  Force 
Base,  FL. 

The  demonstration  was  conducted  from  the  Warton 
Flight  Test  Center  of  the  British  Aerospace  (BAe) 
Military  Aircraft  Division.  The  base  station 
equipment  was  located  at  Warton,  while  missions 
were  flown  over  the  local  area,  the  Irish  Sea,  and 
the  Directorate  of  Test  and  Evaluation  Organization 
(DTEO)  Aberporth  range  facility. 


The  Warton  facility  has  over  50  years  of  successful 
experience  in  the  flight  testing  of  modern  fast  jet 
aircraft  ranging  from  the  Canberra  to  the  current 
EuroFighter.  The  facility  has  a  complete  set  of  test 
mission  support  and  analysis  facilities. 

BAe  has  purchased  a  variant  of  the  RAJPO  system. 
They  will  receive  the  system  later  this  year.  It  will  be  one 
of  a  number  of  instrumented  packages  used  for  testing 
the  weapons  system  for  the  EuroFighter  aircraft.  The 
GPS  instrumentation  will  be  installed  on  as  many  as  four 
chase  Tornado  aircraft  tracked  by  the  sensors  of  the 
EuroFighter.  The  analysts  will  be  able  to  compare  the 
sensor-derived  positions  to  the  GPS-deriv^  TSPI 
positions  of  the  target  aircraft. 

THE  DEMONSTRATION  SYSTEM 

The  key  elements  of  the  RAJPO  system  are  an 
integrated  GPS  P(Y)  Code  Receiver,  Inertial 
Reference  Unit  (IRU),  and  the  Data  Link  System 
(DLS).  The  GPS  receiver  provides  continuous  and 
precise  positioning  data.  The  IRU  aids  the  dead 
reckoning  process  if  GPS  signal  processing  is 
interrupted.  DLS  is  a  communication  and 
networking  system,  designed  by  the  RAJPO,  for  the 
real-time  collection  of  GPS  TSPI  and  platform  data. 
Figure  1  is  an  overview  of  the  system. 

The  TSPI  and  communication  instrumentation 
assets  do  not  interfere  with  the  equipment  under 
test.  As  an  example,  when  in  use  on  high 
performance  fighter  aircraft,  the  equipment  mounts 
in  a  5  inch  diameter  Sidewinder  missile  pod  shell. 
The  pod  connects  to  any  AIM-9  capable  aircraft. 
The  pod  uses  standard  power  available  at  the  AIM- 
9  station.  The  DLS  transmits  the  TSPI  message  to 
the  base  station  for  real-time  analysis  of  the  test 
mission.  To  support  post-mission  reconstruction,  a 
solid-state  memory  system  mounted  in  the  pod 
records  the  measured  TSPI. 
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Figure  1 .  RAJPO  GPS  Receiver-Based  System  Overview 


The  recorded  TSPI  message  is  the  same  message 
that  is  transmitted  to  the  base  station  by  the  DLS. 
The  DLS  processor  subsystem  records  the  mission 
data,  and  passes  this  data  to  the  range  data 
analysis  and  display  system. 

To  minimize  multipath  effects,  the  data  link  antenna 
in  the  pod  focuses  most  of  the  energy  fonvard, 
attempting  to  reduce  the  energy  aft  and  to  the  sides, 
where  multipath  reflections  can  degrade  the  direct 
signal.  The  GPS  antenna  is  a  dual  element 
antenna,  one  element  facing  upward,  while  the 
other  faces  downward.  The  receiver  samples  these 
antennas  at  a  four  millisecond  rate.  The  receiver 
apportions  the  energy  from  each  of  the  antennas 
based  on  the  energy  level  in  each  of  the  antennas 
and  the  orientation  of  the  aircraft. 

The  ground  equipment  consists  of  a  Data  Link 
Controller  and  Processor  subsystem  (DLC/P),  a 
Data  Link  Transceiver  (DLT),  a  GPS  Reference 
Receiver/Processor  (RR/P),  and  an  antenna 
subsystem.  The  base  station  equipment  establishes 
the  network  protocols,  assigns  the  frequency  of 
operation,  assigns  the  time  slots  for  reporting  of 
each  participant,  and  records  all  of  the  mission 
TSPI  data. 

The  GPS  RR  develops  the  differential  GPS 
corrections  and  passes  these  corrections  to  the 
communication  system  for  transmittal  to  the 
exercise  participants.  Alternatively,  the  GPS 
measurements  from  the  participant  receivers  are 


transmitted  to  the  RR/P  for  the  post-mission 
application  of  the  differential  corrections.  Included  in 
the  down-linked  message  are  the  IRU 
measurements.  The  RR/P  processor  combines  the 
differentially  corrected  GPS  measurements  and  IRU 
measurements.  These  measurements  are 
processed  in  the  same  Kalman  filter  that  is  part  of 
the  GPS  receiver  in  the  participant  platform. 

Tests  conducted  at  Eglin  Air  Force  Base  (AFB) 
demonstrate  that  the  accuracy  is  essentially  the 
same  when  either  the  participant  platform  or  the 
base  station  applies  the  differential  GPS 
corrections.  The  decision  whether  to  apply  the 
corrections  in  the  participant  receivers  or  at  the 
base  station  is  an  operational  consideration. 

PRE  DEMONSTRATION  ACTIVITIES 

Two  RAJPO  instrumentation  pods  were  made 
available  for  the  demonstration.  The  base  station 
equipment  was  installed  in  an  interim  trailer,  as 
shown  in  Figure  2.  Figure  3  shows  the  interior  of  the 
van  and  a  portion  of  the  equipment  installed.  It  is  to 
be  noted  that  for  future  utilization,  the  base  station 
equipment  is  being  installed  permanently  in  a  40- 
foot  trailer. 

The  system  provided  for  the  demonstration  is  an 
early  version  of  the  RAJPO  system.  Its  performance 
has  been  validated  on  the  Eglin  AFB  test  range. 
The  RAJPO  system  available  today  takes 
advantage  of  newer  computers,  and  fits  into  one  19- 
inch  rack. 
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Figure  2.  Interim  Trailer,  Exterior  View 


Figure  3.  Interim  Trailer,  Interior  View 


Figure  4.  AS011  Tornado  Aircraft 


The  equipment  was  shipped  from  Eglin  AFB  to  the 
Royal  Air  Force  (RAF)  base  located  at  Mildenhall.  It 
was  driven  to  the  Warton  site.  The  initial  activities 
involved  verification  that  the  equipment  was 
operational,  and  a  check  of  the  weight  and  balances 
to  ensure  compliance  with  the  Interface  Control 
Document  (ICD)  data.  The  pods  were  to  fly  on 
AS011,  a  single  stick  version  of  a  Tornado  Air 
Defense  Variant  (ADV),  and  on  AT001,  a  trainer 
version  of  the  ADV  Tornado  aircraft.  Electro¬ 
magnetic  Compatibility  (EMC)  tests  took  place 
satisfactorily  on  AS01 1 . 

The  base  station  data  link  antenna  was  installed  on 
a  tower  that  was  approximately  10.6  meters  above 
the  ground.  Normally,  the  base  station  uses  a  volute 
antenna  that  provides  coverage  of  360  degrees  of 
azimuth  and  approximately  200  degrees  of  eleva¬ 
tion.  This  antenna  was  not  available  for  the  demon¬ 
stration;  thus,  we  used  a  blade  antenna  that 
focused  most  of  the  energy  upward.  Although  this 
antenna  is  not  optimal,  all  agreed  that  it  would 
suffice  for  the  demonstration.  We  would  expect  the 
data  link  performance  to  suffer  somewhat,  but  not 
enough  to  seriously  affect  the  demonstration.  It  is 
our  understanding  that  when  BAe  installs  the 
system  they  have  purchased,  they  will  locate  the 
DLS  antenna  at  a  higher  location  than  the  10.6 
meter  tower  used  for  the  demonstration. 

DEMONSTRATION  FLIGHTS 

The  initial  flight  took  place  on  28  February  1996 
with  aircraft  AS01 1  as  shown  in  Figure  4. 

As  shown  in  Figure  5,  we  made  the  test  as 
“unfriendly”  as  possible  by  mounting  the  pod  to  the 
inboard  stub  of  the  inboard  pylon,  nearest  the 
fuselage  and  inboard  to  a  2250  liter  fuel  tank.  This 
location  would  create  the  highest  possibility  of 
signal  blockage  between  the  pod  antenna  and  the 
satellites,  and  between  the  pod  antenna  and  the 
base  station. 

The  demonstration  was  to  take  place  over  the  Irish 
Sea,  west  of  the  Warton  site  complex.  Prior  to 
initiating  the  demonstration,  there  was  a  commit¬ 
ment  for  the  aircraft  to  travel  to  the  DTEO  range  at 
Aberporth,  approximately  160  kilometers  to  the 
South  of  Warton.  We  decided  to  evaluate  the 
performance  of  the  system  during  this  activity. 

The  transmitted  power  from  the  transceiver  in  the 
pod  and  the  normal  antennas  used  in  the  pod  and 
base  station,  theoretically,  limits  reliable  operation 
up  to  distances  of  115  kilometers.  In  operation,  we 
typically  observe  reliable  data  link  operation  on  DoD 
ranges  at  distances  of  250  kilometers  between 
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Figure  5.  AS011  Aircraft  with  RAJPO  Pod  at  the 
inboard  Station 


participants  and  the  ground  station.  As  previously 
described,  we  did  not  expect  the  antenna  used  at 
the  base  station  to  limit  performance,  but  we  could 
not  be  sure  until  the  demonstration  took  place. 

The  data  link  performed  competently,  and  we  did 
receive  valid  TSPI  data  during  the  two  hours  the 
aircraft  was  at  Aberporth.  While  at  Aberporth,  the 
aircraft  did  a  number  of  maneuvers.  When  the  pod 
was  between  the  fuselage  and  the  base  station 
antenna,  message  integrity  exceeded  95%.  When 
the  fuselage  was  between  the  pod  and  the  base 
station  antenna,  message  integrity  was  slightly 
better  than  65%,  a  number  that  is  adequate  for 
many  applications. 

However,  to  assure  message  integrity  in  any 
platform  orientation,  and  at  long  distances  between 
the  aircraft  and  base  station,  RAJPO  can  supply 
remote  data  link  transceiver  stations.  The  remote 
station  will  relay  the  received  aircraft  messages  to 
the  base  station.  To  assure  adequate  coverage 
during  a  test  mission,  many  of  these  remote 
stations  are  usable  on  a  range.  The  base  station  will 
combine  messages  received  directly  from  the 
participating  aircraft  with  the  messages  relayed 
from  all  of  the  remote  sites,  further  enhancing  the 
message  reliability  of  the  system. 

The  remote  station  includes  a  DLS  transceiver,  an 
interface  unit,  power  conditioner,  and  an  optional 
rubidium  frequency  standard.  Since  they  are  often 
exposed  to  the  elements,  the  assemblies  are  in  an 
environmentally  secure  enclosure.  These  remote 
transceiver  units  measure  approximately  483  x  711 
X  610  mm,  and  are  easily  relocated.  The  remote 
stations  are  often  located  in  areas  that  are  not 
easily  accessible.  Hence,  an  additional  concern  is 
power  to  operate  these  units.  Some  of  the  US 


ranges  operate  these  remote  stations  with  solar 
power  subsystems. 

A  remote  transceiver  station  positioned  to  the  south 
of  Aberporth  would  have  maintained  the  message 
integrity  of  95%,  regardless  of  aircraft  orientation. 
The  RAJPO  specification  states  a  message 
reliability  of  99%  when  the  participant  pod  can 
maintain  line  of  sight  to  two  ground  stations. 

RAJPO  provides  software  to  determine  the  optimal 
location  of  data  link  transceiver  sites  for  any  defined 
mission  and  topography.  Using  this  software,  the 
Range  Manager  can  “fly”  the  aircraft  through  the 
range  to  determine  the  expected  data  link  and  GPS 
performance,  and  then  locate  remote  site(s)  for 
optimal  coverage  during  a  mission. 

When  the  aircraft  completed  its  mission  over  the 
Aberporth  range,  it  traveled  to  the  demonstration 
area  over  the  Irish  Sea,  approximately  50 
kilometers  from  the  Warton  base  station.  The 
aircraft  flew  a  variety  of  relatively  low  dynamic  flight 
profiles,  including  race  track  turns  and  ‘S’  turns.  We 
can  expect  the  maximum  amount  of  multipath  when 
flying  over  sea  water  at  low  altitudes.  However,  the 
aircraft  flew  at  an  altitude  of  approximately  160 
meters  over  the  Irish  Sea  without  any  degradation 
of  message  integrity  or  TSPI  accuracy. 

During  all  of  the  demonstration  period  over  the  Irish 
Sea,  data  link  integrity  was  in  excess  of  95%,  and 
the  TSPI  data  was  of  fine  quality  in  all  aircraft 
orientations.  It  did  not  matter  whether  the  pod 
antenna(s)  was  facing  the  base  station,  or  if  the  pod 
antenna(s)  line  of  sight  to  the  base  station  was 
blocked  by  the  aircraft  fuselage  or  fuel  tank. 

A  two  pod,  two  aircraft  demonstration  took  place  on 
1  March.  To  provide  as  much  diversity  as  possible 
to  the  demonstration,  one  pod  was  located  on  the 
left  wing  of  AS01 1  and  the  other  on  the  right  wing  of 
AT001.  Once  again,  the  pods  were  connected  to  the 
inboard  stub  of  the  inboard  pylon  of  each  aircraft. 
As  before,  the  aircraft  flew  to  the  Aberporth  range 
prior  to  returning  to  the  demonstration  area  over  the 
Irish  Sea.  We  were  able  to  receive  reliable  (94.5%) 
TSPI  data  from  the  two  aircraft  while  they  were  at 
the  Aberporth  range,  and  on  their  return  to  the 
demonstration  area  over  the  Irish  Sea.  The  reliable 
TSPI  data  resulting  from  this  activity  continues  to 
demonstrate  the  high  performance  of  the  RAJPO 
system. 

Using  the  two  aircraft  participating  in  this  exercise, 
we  were  able  to  demonstrate  the  automatic  relay 
capability  of  the  DLS,  i.e.,  the  DLT  continuously 
seeks  the  most  efficient  routing  for  a  participant 
message  back  to  the  base  station.  This  automatic 
routing  feature  of  the  RAJPO  system  uses  other 
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aircraft,  ground  based  remote  DLS  sites,  as  relay 
nodes.  The  DLS  will  support  up  to  five  levels  of 
relay  for  the  transmission  of  a  single  message.  We 
were  able  to  successfully  demonstrate  the  aircraft- 
to-aircraft-to-base  station  relay  capability  of  the 
system. 

On  4  March  the  exercise  was  repeated  success¬ 
fully.  On  7  March  a  final  demonstration  took  place. 
On  this  day,  we  flew  one  pod  on  each  wing  of  the 
AS01 1  aircraft.  The  aircraft  did  not  have  to  travel  to 
Aberporth.  We  had  two  hours  of  successful  flying 
time,  repeating  the  fine  performance  observed 
during  the  previous  tests. 

CONCLUSION 

The  demonstration  of  the  performance  of  the 
RAJPO  system  was  an  unqualified  success.  The 
RAJPO  assets  demonstrated  that  they  can  support 
a  wide  range  of  flight  test  activities.  The 
demonstration  allowed  BAe  to  gain  valuable 
information  about  the  installation  and  use  of  the 
variant  system  they  will  receive.  BAe  will  install  the 
base  station  equipment  permanently  at  the  Warton 
ground  station  Command  Center.  The  master  DLS 
antenna  wiil  mount  on  a  tower  that  is  much  higher 
than  the  location  used  during  the  demonstration.  A 
remote  DLS  transceiver  installation  site  will  improve 
the  coverage  for  anticipated  EuroFighter  flight 
testing  of  the  onboard  weapon  system.  Both  the 
master  and  remote  stations  will  use  volute  antennas 
rather  than  the  blade  antenna  used  in  the 
demonstration.  These  factors  should  result  in  a 
system  performance  that  exceeds  that  observed 
during  this  demonstration. 


Figure  6.  The  Team  Bringing  the  Success 


Figure  6  shows  the  entire  team  that  supported  this 
demonstration  activity.  Only  as  a  result  of  the  total 
support  of  all  the  members  of  the  team  were  we 
able  to  successfully  execute  the  demonstration  of 
the  system. 
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Abstract 

The  need  to  optically  track  fast  moving  targets 
with  great  accuracy  has  always  been  a  prime 
objective  of  Test  and  Evaluation  Ranges. 
During  the  Second  World  War  the  kinematic 
theodolite,  commonly  known  as  the 
kinetheodolite  was  developed  for  this  purpose 
and  although  many  improvements  have  been 
made  since  then,  the  principles  of  operation 
have  not  significantly  changed. 

As  defence  technology  advanced,  new,  more 
challenging  demands  were  placed  upon  optical 
tracking  instrumentation.  Targets  became  more 
dynamic  and  complex  and  customers  soon 
started  calling  for  greater  degrees  of  accuracy, 
faster  image  frame  rates  and  improved  image 
quality. 

These  new  requirements  eventually  prompted  a 
feasibility  study  to  investigate  a  replacement 
for  the  kinetheodolite  and  this  in  turn  led  to  the 
development  of  the  modern  day  Electro-Optical 
Tracking  Instrument  (EOTI). 

These  mobile  instruments  are  capable  of 
automatically  tracking  small,  very  fast  moving 
objects  at  increased  ranges.  They  have 
excellent  angular  resolutions  and  their  multi¬ 
sensor  platforms  are  capable  of  high  rates  of 
angular  acceleration  and  the  video  frame  rates 
necessary  to  meet  the  demands  of  present  and 
future  customer  requirements. 

This  paper  presents  a  historical  case  study  of 
the  changes  in  optical  tracking  within  Ranges 
concentrating  upon  the  evolution  of  the  early 
kinetheodolite  into  the  modern  day  and  future 
EOTI. 


Introduction 

Why  do  we  need  Test  Ranges?  The  answer  to 
this  question  is  we  need  to  prove  operational 
capability  of  weapon  systems,  provide 
empirical  evaluation  of  performance  in  terms 
of  stated  design  objectives  and  to  provide 
information  that  may  allow  customers  to 
improve  the  performance  envelopes  of  their 
systems. 

Generally  for  tracking  instruments,  this  means 
determination  of  trajectory,  behaviour  and 
performance  of  in-flight  projectiles  whether 
they  be  shells,  missiles,  flares,  bombs,  sonar- 
buoys  or  aircraft. 

Over  the  years,  measurement  techniques  have 
needed  to  advance  in  order  to  cope  with  the 
demands  placed  upon  them  by  rapidly 
developing  weapon  systems. 

Prior  to  the  Second  World  War,  the  only 
ground  based  metric  techniques  utilised  to 
precisely  determine  the  position  of  spatial 
objects  were  simple  bomb  scoring  theodolite 
systems. 

In  this  system,  a  pair  of  manually  directed 
theodolites  sited  for  good  triangulation 
geometry  followed  the  target  with  post-mission 
target  information  being  derived  from  recorded 
film  records. 

In  America,  the  Akely  Cinetheodolite  was 
developed  by  the  Mitchell  Camera  Corporation 
while  in  Germany,  Askania-Werke  produced 
the  Kinetheodolite. 

The  former  employing  a  cine  film,  intermittent 
movement  and  the  latter  a  pulse  film  movement 
while  the  Askania  had  the  superior 
performance  characteristics. 
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Fig  1.  A  modified  Askania  kinetheodolite. 


With  the  advent  of  Radar  during  the  Second 
World  War,  automatic  tracking  of  spatial 
objects  became  widely  adapted  for  Weapon 
Ranges  work.  It  provided  readily  available 
target  positional  data  in  visible  and  often  non- 
visible  conditions,  but  the  accuracy  was  in  the 
region  of  ten  times  lower  than  that  achievable 
using  a  kinetheodolite  system. 

Tracking  radars  were  used  during  the  early 
missile  development  era  to  slave  direct  servo 
platforms,  usually  modified  anti-aircraft  gun 
mounts  carrying  high-speed  cine  cameras.  This 
technique  enabled  much  higher  spatial 
positional  accuracies  to  be  derived. 

The  Early  Years 

A  variety  of  optical  instrumentation  has  been 
developed  for  target  tracking  over  many  years. 
Some  of  the  more  common  types  are: 

i)  Kinetheodolites 

ii)  Ballistic  cameras 

iii)  High  speed  tracking  cameras 

iv)  Automatic  TV  trackers 

Soon  after  the  end  of  the  Second  World  War, 
most  western  Missile  Test  Ranges  acquired 
numbers  of  wartime  German  Askania 
kinetheodolites,  and  these  with  a  certain 
amount  of  modification  have  supplied  an 
excellent  service  for  many  years.  However, 
they  had  their  disadvantages. 

These  limitations  which  in  turn  prompted 
major  developments  may  be  classed  under  four 
headings: 


i)  Accuracy 

ii)  Data  rate 

iii)  Target  acquisition  difficulties 

iv)  Data  availability  time 

The  generally  accepted  accuracy  of  an 
analogue  Ascania  theodolite  in  an  ideal  two 
instrument  tracking  system  is  about  15  seconds 
of  arc  rms  giving  positional  accuracies  in  the 
region  of  0.6  m  at  a  range  of  1500  m  increasing 
to  12  m  at  a  range  of  20  Km. 

While  this  was  initially  acceptable  for  short- 
range  work,  pressure  for  improved  accuracies 
grew  with  the  extra  precision  required  when 
measuring  parameters  such  as  position, 
velocity,  acceleration,  pitch,  roll,  yaw  and 
miss-distance. 

Another,  very  important  reason  for  having 
more  precise  optical  systems  was  the  need  to 
offer  a  measurement  standard  to  allow  other 
instrumentation  to  assess  and  maintain 
performance. 

These  requirements  led  directly  to 
developments  such  as  the  Askania  GIGAS  and 
Contraves  Electro-Optical  Tracker  which 
offered  individual  instrument  accuracies  in  the 
region  of  0.0014  degrees  rms. 

The  demand  for  more  precise  velocity  and 
acceleration  measurements  also  led  to  required 
increases  in  exposure  rates  and  in  some  cases 
mechanisms  were  enhanced  to  achieve  30 
fi-ames  /  second. 


Fig  2.  High  speed  tracking  camera 


Although  increased  fi’ame  rates  offered  greater 
time  based  resolution,  there  was  a  disadvantage 
in  increased  amounts  of  data  to  collect,  process 
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and  analyse,  drastically  increasing  the  data 
availability  time. 

A  major  problem  remained  with 
kinetheodolites  still  requiring  an  operator  in  the 
servo  loop  in  order  to  acquire  and  track  the 
target.  It  is  known  to  be  difficult  to  acquire 
targets  travelling  towards  or  away  from  the 
instrument  and  difficult  to  track  a 
manoeuvring,  agile  target. 

Various  aids  were  developed  which  split 
broadly  into  two  types,  namely  acquisition 
indicators  and  servo  drives. 

Both  systems  accepted  data  from  some 
independent  source  and  converted  this  pointing 
data  to  a  polar  form  with  local  origins  set  at  the 
particular  instrument  to  be  aided. 

This  data  could  then  either  be  displayed  as  a 
deflection  in  the  eyepiece  of  the  operator  or 
used  to  drive  servomotors,  pointing  the 
instrument  towards  the  target. 

Single  operator  kinetheodolites  also,  on 
occasions  suffered  complete  loss  of  target  track 
due  to  erratic  operator  movements. 

An  operator  without  having  to  change  stance  or 
move  the  feet  can  normally  traverse  tracking 
arcs  of  up  to  25  degrees  in  azimuth.  When 
greater  operator  movements  occurred,  real  time 
data  degradation  was  often  the  result. 

Two-man  operation,  one  directing  each  axis 
eased  this  problem  but  such  instruments  tended 
to  be  expensive. 

With  the  introduction  of  smaller,  more  agile 
weapons,  the  demand  upon  operator 
performance  increased  together  with  the 
requirement  for  yet  higher  accuracies,  the  need 
for  target  attitude  information  and  the  regular 
requirement  to  track  specific  physical  features 
of  the  target.  This  led  to  the  first  developments 
of  auto-tracking  platforms  and  the  eventual 
integration  of  a  video  camera  into  the 
kinetheodolite 

This  idea  involved  an  electronic  auto-tracking 
unit  accepting  video  and  crudely  calculating 
what  it  considered  to  be  a  target.  It  then 
produced  error  signals  to  the  servomotors  in 
order  to  keep  the  target  in  the  centre  of  the  field 
of  view.  Another  advantage  was  the  video  error 


could  be  summed  with  main  scale  values  to 
produce  real  time,  high  precision  pointing  data. 

Another  early  development  in  the  automatic 
optical  tracker  field  was  the  laser  tracker, 
LIDAR  and  the  American  counterpart  PATS. 
These  worked  well  with  large,  co-operative 
targets  such  as  aircraft,  but  many  missiles  were 
deemed  unsuitable  platforms  for  mounting  the 
necessary  large  reflectors  due  to  aerodynamic 
and  other  problems. 

Lasers  became  very  useful  as  target  ranging 
devices  although  their  eye  safety  problems 
prohibited  wide  spread  use.  Today,  high  power 
pulsed  eye-safe  lasers  are  available  for 
integration  into  any  tracking  system. 

By  the  early  seventies,  the  disadvantages  of 
then  currently  available  optical  and  electronic 
instrumentation  was  of  great  concern  to  UK 
weapon  development  contractors.  There  were 
new  requirements  for  precision  tracking  of 
multiple  sub-munitions  and  rapid  data  turn 
around  times.  Contractors  were  given  tight 
development  time  scales  and  offered  incentives 
to  achieve  them.  This  forced  Test  Ranges  to 
once  again  reassess  their  tracking 
instrumentation. 

The  Instrumentation  Department  at  RAE 
Famborough  was  tasked  to  investigate  and 
develop  new  precision  tracking  instruments  for 
UK  Ranges.  The  investigation  led  to  two 
distinct  options. 

i)  To  procure  new  digital  kinetheodolites 
with  integral  TV  trackers  and  combine 
the  data  produced  with  radar  data  into 
some  form  of  high-speed  data  bus. 

ii)  To  develop  mobile  servo  platforms 
which  could  carry  multiple  sensors, 
optical  and  electronic.  These  platforms 
must  also  be  capable  of  working  as  a 
single  trajectory  measurement  station 
and  /  or  be  incorporated  into  a  Range 
data  bus. 

The  latter  option  was  favoured  and  the  basic 
specifications  were  agreed  to  be: 
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i)  a  single  station  trajectory  measuring 
system 

ii)  tracking  accuracies  to  be  better  than  50 
micro-radians  la  in  angle  and  1  metre 
in  range. 

iii)  data  rate  selectable  up  to  a  maximum  of 
100  samples  /  second 

iv)  with  a  full  five  sensor  payload,  must  be 
capable  of  angular  slew  and 
acceleration  rates  of  1 00°/  second  and 
100°/  second /second  respectively 

v)  compact  onboard  recording  of  all  sensor 
images,  angular  data,  range,  status  and 
timing  information 

vi)  built  in  self  calibration  system 

vii)  individual  systems  must  be  completely 
mobile 

viii)  real-time  data  output  interface  for  a 
standard  data-bus 

ix)  capable  of  being  operated  and 
maintained  by  two  operators 

IT6,  Famborough  procured  a  Swedish,  Saab- 
Scania  TLT-510  without  it  laser  range-finder 
for  evaluation  and  subsequent  development 
into  a  prototype  Electo-Optical  Tracking 
Instrument  (EOTI). 


Fig  3.  The  Saab-Scania  TLT-51 0 

During  the  TLT-510  manufacturing  period,  IT6 
department  carried  out  a  comparative  study  of 
millimetric  radar  and  laser  ranging  devices. 
From  this  study,  it  became  clear  that  in  order  to 
achieve  a  range  accuracy  of  0.5  metres  at 
extended  ranges,  both  methods  required  some 
form  of  co-operative  device  fitted  to  the  target. 
The  only  laser  then  capable  of  generating 
sufficient  pulse  power  for  long-range  tracking 


was  the  Nd  Yag,  a  non-eye  safe  laser  operating 
at  1 .06  micrometers  wavelength. 

Due  to  safety  and  other  problems  associated 
with  the  laser,  a  millimetric.  Range  Only  Radar 
(ROR)  was  developed  with  the  help  of  RRE 
Malvern  with  the  prototype  production  contract 
being  awarded  to  MCCS  of  Frimley. 


Fig  4.  The  TLT-510  with  prototype  radar 


This  prototype  ROR  was  mechanically 
integrated  into  the  TLT-150  and  the  complete 
system  mounted  on  a  modified  Rapier  trailer 
for  the  purpose  of  field  tests. 

The  prototype  EOTI  was  then  tested  for 
accuracy  using  drone  targets  fitted  with 
millimetric  Lunarberg  lenses.  Tests  were 
carried  out  at  the  Aberporth  Range  and  range 
accuracies  of  0.3  metres  were  consistently 
achieved  against  Jindivik  targets  out  to  Ranges 
of  20  Km. 

Three  years  of  EOTI  testing  as  a  single  station 
system  followed  with  acquisition  assistance 
derived  from  tracking  radar  via  microwave 
links.  These  tests  took  place  at  the  three  RAE 
test  Ranges  of  Aberporth,  Larkhill  and  West 
Freugh  against  a  variety  of  targets  including 
aircraft,  bombs  and  land,  air  and  sea  launched 
missiles.  Accuracy  performance  and  data  rates 
all  met  the  set  requirements,  but  the  narrow 
beam  width  of  the  optics  introduced  major 
target  acquisition  problems,  especially  against 
high  speed,  dynamic  targets. 

Various  acquisition  options  were  then  studied 
including;  zoom  optics,  target  prediction  and 
moving  target  detection.  Eventually,  a 
combination  of  all  three  techniques  proposed 
by  British  Aerospace,  Bracknell  offered  the 
greatest  promise. 
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In  1 984,  a  Ranges  committee  agreed  the  final 
EOTI  technical  specification  and  production 
tenders  were  requested  from  industry, 
worldwide. 

Bids  were  received  from  eight  firms,  three 
British,  three  French,  one  Swiss  and  one 
American.  At  the  same  time,  MCCS  were 
awarded  a  production  contract  for  12  ROR’s. 
The  tenders  called  for  seven  general-purpose, 
four  high  precision  and  two  special  EOTI’s  that 
were  to  be  dedicated  to  and  paid  for  by  one 
project  office. 

Contraves  Goerz  offered  thirteen  identical 
servo  platforms  with  optics  and  electronics  to 
satisfy  all  three  types  of  EOTI.  British 
Aerospace,  Bracknell  offered  a  similar  solution 
but  employing  German  MBB  platforms,  Zeiss 
optics  and  British  Aerospace  control 
electronics. 

Contraves  offered  eleven  SkyTrack  instruments 
and  two  of  Goerz’ s  servo  platforms  for  the 
special  requirement. 

After  careful  consideration  of  all  the  options, 
British  Aerospace,  in  partnership  with  MBB  of 
Bremen,  ERG  were  awarded  the  contract  for 
the  EOTI’s  and  MCCS  of  Frimley  were 
awarded  the  contract  to  manufacture  the  Range 
Only  Radars  which  were  to  be  integrated  into 
the  EOTI’s. 


Present  day  technologies 
The  EOTI 

The  present  day  EOTI’s  are  high  precision, 
auto-tracking  instruments  capable  of  carrying  a 
large  payload  of  various  sensors,  including 


video  cameras,  Range  Only  Radar,  InffaRed, 
and  High-Speed  cameras. 

The  EOTI  supplied  by  British  Aerospace 
Plymouth  and  MBB  comprises: 

i)  The  MSP  2000,  servo  platform  fitted 
with  the  MCCS  Range  Only  Radar  type 
282,  video  Cameras,  film  cameras  and 
associated  optics. 

ii)  A  British  Aerospace  TV  auto¬ 
acquisition  and  auto  tracking  system. 

iii)  A  Control  and  Record  Module  (CRM) 
which  also  communicates  with  other 
Range  instrumentation. 

The  MSP-2Q0Q  EOTI  servo  platform. 

The  MSP-2000  precision  platform  is  a  ‘T’ 
mount  configuration  capable  of  carrying  up  to 
seven  sensors  at  a  weight  of  up  to  1000  kg.  It 
has  angular  velocity  and  acceleration 
capabilities  of  100°/second  and  100°/  second 
/second  respectively.  It  is  carried  on  a  heavy- 
duty  trailer  and  is  operated  remotely  via  a  fibre 
optic  link  from  the  CRM.  The  platform 
basically  comprises  a  pedestal  azimuth  unit, 
elevation  unit  and  a  pair  of  sensor  attachment 
mounts.  Both  elevation  and  azimuth  shafts  are 
driven  by  Samarium  Cobalt  DC  torque  motors 
in  direct  drive  configuration.  Optical  encoders 
attach  directly  to  the  Azimuth  and  elevation 
shafts  and  measure  angular  position  to  a 
resolution  of  21  bits  (0.00017  degrees). 

The  trailer  with  road  suspension  and  rear 
electronics  compartment  consists  of  a  rigid 
central  chassis,  to  which  the  platform  is 
connected  via  a  three-point  fine  leveling 
system. 

When  in  operation,  the  centre  chassis  is 
supported  by  four  adjustable  jacks,  used  for 
course  leveling  of  the  instrument. 

During  any  trial,  video  information  together 
with  target  line  of  sight  angles,  target  range  and 
synchronised  time  is  recorded  to  videotape.  All 
measurement  and  status  digital  information 
generated  by  the  EOTI  is  also  recorded  through 
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Fig  5.  A  British  Aerospace/MBB  EOTI  mount 


an  industrial  standard  interface  to  magneto¬ 
optical  disk  I 

A  flat-top  vehicle  carries  the  Control  and 
Record  Module  and  tows  the  servo  platform 
trailer.  During  deployment,  the  vehicle  and  the 
trailer  are  separated  and  when  not  in  use,  the 
platform  is  protected  from  the  elements  by  a 
clam-shell  type,  hydraulically  operated  cover. 


EOTI  outline  specification 

The  capabilities  of  the  instrument  are 
summarised  below. 


EOTI  sensors 

A  wide-angle  acquisition  camera  is  provided  to 
assist  the  operator  in  acquiring  agile  targets. 
Once  acquired,  the  automatic  tracking  loop  is 
closed  via  a  mechanically  shuttered,  high 
resolution  video  camera  and  a  British 
Aerospace  auto-tracking  unit.  The  main  Multi- 
Focal  Lens  has  selectable  fields  of  view 
(horizontal)  of  1.4°,  0.7°  and  0.35° 
corresponding  to  focal  lengths  of  0.5m,  Im  and 
2  m  respectively. 

IR  sensors  and  an  additional  telescope  is 
provided  for  mating  with  a  high-speed  film 
camera. 


Angular  velocity 

100°/second 

Angular  acceleration 

1 00°/second  /second 

Maximum  payload 

1 000kg  (balanced) 

Tracking  lens 

Multi-Focal  length 

0.5ni/1.0m/2.0m 
fll  at  2m 

Tracking  accuracy 

20  microradians  best 

Mechanical  accuracy 

15  microradians  best 

ROR  accuracy 
(point  target) 

0.5m  <10km 
0.005%range  >10km 

ROR  range  rate 
measurement  accuracy 

1.5m/s<10km 
proportionately  greater 
for  ranges  greater  than 
10km 

RORPRF>^' 

8kHz  <1 4km 

4kHz  >14km,  <3  5km 

2  kHz  >  35km 

ROR  Tracking  Range 

Missile  typically  10km 
Aircraft  typically  35km 

In  order  to  achieve  the  required  dynamic 
tracking  accuracy  of  worst  case  50 
microradians  or  0.0029  degrees,  the  instrument 
needs  to  be  located  on  a  highly  stable  plinth.  It 
is  possible  however  to  deploy  an  instrument  on 
an  un-prepared  surface  with  some  degradation 
in  angular  accuracy. 


Fig  5.  A  British  Aerospace/MBB  EOTI  mount 


EOTI  Control  and  Record  Module  (CRMI 

The  EOTI  is  controlled  remotely  by  an  operator 
in  the  Control  and  Record  Module,  a 
transportable  cabin  based  upon  an  international 
style  shipping  container  which  is  compliant 
with  tempest  requirements.  This  cabin  is  linked 
to  the  servo-platform  via  a  multiple  fibre-optic 
cable.  The  CRM  contains  the  following  main 
components: 

i)  EOTI  control  and  display  console 

ii)  System  control  and  display  computers 

iii)  Digital  and  video  data  recorders 

iv)  VDU  for  post  mission  analysis 

v)  Interface  computer 
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EOTI  video  ima2e  auto-tracker 

At  the  heart  of  the  auto-tracking  loop  lies  the 
British  Aerospace  auto-tracking  unit  which  is 
capable  of  automatically  acquiring  up  to  5 
moving  targets  appearing  within  the  tracking 
video  camera  field  of  view  and  subsequently 
auto-tracking  any  two  of  these  simultaneously. 
Using  centroid,  correlator  or  other  tracking 
techniques,  target-to-sensor  boresight  errors  are 
accurately  estimated  for  each  target.  For  one 
selected  target,  these  errors  are  used  to  drive 
the  servo  platform  via  a  closed  loop  to  align  the 
sensor  boresight  with  the  target. 


EOTI  operation 

On  fully  instrumented  test  Ranges,  target 
acquisition  assistance  is  made  available  from 
any  of  the  following  systems;  surveillance 
radar,  tracking  radar  or  other  EOTI’s. 


Fig  6.  EOTI  lorry,  CRM  and  trailer  ready  for 
deployment. 


Other  optical  instrumentation 

Other  tracking  and  non-tracking  optical 
instrumentation  still  used  on  DERA  Ranges 
include: 

i)  Pin  registered  16mm  Photosonic  and 
NAC  high-speed  cameras  capable  of 
operating  at  speeds  up  to  1 000 
frames/second. 

ii)  Steerable  35  mm  tracking  used  for 
behavioral  analysis. 

iii)  Fixed  35mm  launcher  cameras  running 
at  a  phase  locked  100  frames/second. 


iv)  Fixed  70mm  cameras  running  at  100 
frames/second  generally  used  for  high 
definition  work. 

v)  Vinten  10/21  type  tracking  cameras. 
Generally  mounted  on  other 
instrumentation  such  as  EOTl's. 

The  present  day  kinetheodolite 

There  are  in  the  order  of  24  functional 
kinetheodolites  still  providing  a  valuable 
function  for  DERA  Ranges  and  are  mainly 
used  for  short-range  tracking  trials.  As  stated 
previously,  they  have  remained  relatively 
unchanged  over  the  years  with  some  being 
modified  more  than  others.  The  main 
enhancements  include  10  frame  /  second 
shutter  mechanisms,  the  integration  of  the 
acquisition  aid  systems,  electronic  flash  type 
azimuth  and  elevation  exposure  and  various 
lens  modifications. 

To  continue  using  these  instruments 
indefinitely  poses  many  problems  associated 
with  high  maintenance  costs,  high  film  and 
processing  costs  and  high  infrastructure  costs. 
Other  factors  include  a  reduction  in  short 
range,  ground  launched  firings  and  a  relative 
increase  in  long  range  work. 

A  combination  of  these  two  factors  has  led  to  a 
gradual  reduction  in  the  use  of  kinetheodolites 
to  the  extent  where  today,  only  5%  of  data  used 
for  analysis  is  colleeted  using  the 
kinetheodolite. 

It  is  DERA’s  intention  to  begin  to  phase  out 
kinetheodolites  at  the  end  of  1998.  This  will  be 
made  possible  by  investments  into  a  new 
Mobile  Video  Tracking  (MVT)  system, 
development  of  new  EOTI  sites,  effective  trials 
programming  and  increased  campaigning  of 
already  mobile  instrumentation. 

One  demand  for  the  kinetheodolite  worth 
mentioning  is  the  interest  expressed  by  various 
museums  in  the  UK  and  Europe.  A  type  42 
Askania  kinetheodolite  will  be  donated  to  a 
museum  in  PeenemOnde.  A  later  type  52  and  a 
HS300  high-speed  camera  have  been  donated 
to  another  interested  museum  in  Birmingham, 
UK. 
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The  Future 

The  need  to  modernise  instrumentation  and 
stay  ahead  of  the  game  is  paramount  as  new 
technologies  invariably  result  in  enhanced 
performance  at  reduced  cost  leading  to  greater 
efficiency  and  added  customer  value. 

The  move  towards  digital  imaging 

Photographic  film  has  been  used  extensively 
for  many  years  within  Test  and  Evaluation 
Ranges,  it  is  however  expensive  in  terms  of  the 
cost  of  both  materials  and  processing.  Another 
significant  disadvantage  is  the  processing  time 
involved,  adding  delay  to  the  critical  data  turn¬ 
around  time. 

Video  has  now  been  widely  used  for  a  decade 
or  so  in  gathering  optical  data  and  although 
presently  lacking  in  terms  of  resolution  when 
compared  to  photographic  mediums,  still  has 
much  to  offer. 

The  area  of  digital  video  is  advancing  at  quite  a 
rate  both  in  terms  of  resolution  and  achievable 
frame  rates  and  can  currently  be  used  as  a 
direct  replacement  for  much  of  the 
photographic  media  used  on  Ranges  today.  The 
Optical  Instrumentation  facility  is  looking  very 
closely  at  this  technology  and  its  possible 
applications  and  associated  cost  savings. 

Mobile  Video  Tracking  system  (MVTl 

The  Mobile  Video  Tracking  (MVT)  system  is 
an  area  of  investment  that  will  be  instrumental 
in  the  eventual  phasing  out  of  kinetheodolites. 
This  proposed  system,  consisting  of  at  least 
three  individual  instruments  will  meet  a 
customer  need  in  terms  of  the  relatively  short 
range  tracking  of  a  variety  of  targets,  ranging 
from  dynamic  missiles  to  large  cargo  drops. 
The  instruments  will  be  flexible  in  terms  of  the 
instrumentation  they  will  carry  and  they  will  be 
highly  mobile,  transportable  using  existing 
Range  vehicles. 

Although  the  final  specification  for  this  system 
is  not  yet  finalised,  individual  instrument 
angular  accuracies  will  be  in  the  region  of  0.01 
degrees  or  better. 


Future  EOTl’s 

The  EOTI,  although  a  very  capable  instrument 


as  it  stands,  will  require  enhancement  in 
coming  years.  The  main  areas  of  investment 
are  anticipated  to  be: 

i)  Improved  tracking  ability  achievable 
through  investment  in  modem,  highly 
configurable  auto-tracking  techniques. 
This  would  allow  Improved  acquisition 
of  increasingly  agile  targets  and  a  better 
chance  of  success  in  tracking  the 
unpredictable  target. 

ii)  Enhanced  imaging  through  the  use  of 
new  data  recording  mediums,  low  noise 
video  components  and  mastering 
techniques. 

iii)  Improved  flexibility  in  meeting 
customer  requirements  through  the  use 
of  flexible  acquisition  and  tracking  lens 
assemblies. 

iv)  Improved  radar  range  through 
investment  in  new,  low  noise  front-end 
amplifiers. 

Infra-Red  (IR) 

IR  cameras  are  essential  when  tracking  is 
required  in  poor  visibility  or  when  light  levels 
are  low.  Although  Ranges  have  an  IR 
capability  in  the  form  of  several  far-band  IR 
cameras,  the  use  of  near  band  infra-red  has 
been  found  to  be  of  great  use  when  observing 
small,  rapidly  moving  projectiles. 
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Conclusion 

This  paper  has  chartered  the  evolution  of 
optical  tracking  instrumentation  within  British 
Test  Ranges  from  the  simple  bomb  scoring 
theodolites  to  the  modem  day  and  future 
Electro-Optical  Tracking  Instmment.  This 
development  has  involved  the  growth  of  a 
variety  of  technologies  such  as  the  early 
kinetheodolite  offering  accuracies  of  typically 
0.004  degrees  and  modified  kinetheodolites 
offering  aided  acquisition,  increased  framing 
rates  and  improved  lens  assemblies. 
Radar-slaved  servo  platforms  followed  with 
human  input  being  removed  from  the  control 
loop  for  the  first  time.  This  idea  was  further 
developed  with  the  advent  of  laser  and  video 
auto-tracking  and  auto  acquisition  systems. 
Studies  in  the  early  1980's  led  to  the 
procurement  of  a  SAAB  TLT-510  servo- 
platform  which  in  turn  paved  the  way  for  the 
final  specification  and  eventual  production  of 
the  present  day  EOTI. 

The  need  to  modernise  and  remain  capable  of 
meeting  present  and  future  customer 
requirements  remains  paramount.  Exploring 
and  adopting  new  technologies  is  necessary  in 
achieving  enhanced  performance  and 
invariably  results  in  greater  efficiency  and 
added  customer  value. 

Improvements  in  Optical  Instrumentation  are 
very  definitely  customer  orientated  with 
planned  EOTI  enhancements  in  the  areas  of 
adaptable  lens  configurations,  improved  Range 
Only  Radar,  improved  IR  capabilities  and 
enhanced  auto-tracking  capabilities. 

The  vision  is  ultimately  to  be  able  to  supply  the 
customer  with  the  required  high  quality  results, 
quickly,  first  time,  every  time  with  minimum 
preparation  and  practice. 

Demands  facing  Optical  Instrumentation 
involve  observing  more  complex  and  dynamic 
targets,  requiring  extensive  behaviour  and 
trajectory  information  often  for  a  minimal 
financial  commitment.  New,  compact  and 
highly  Mobile  Video  Tracking  instrumentation 
is  planned,  initially  taking  over  the  roll  of  the 
kinetheodolite,  but  quickly  extending  that  role 
in  order  to  maintain  and  attract  new  business. 


essential  in  the  future  success  of  DERA  T&E 
Ranges. 
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1 ,0  Abstract 

After  the  formation  of  BMW  Rolls-Royce  in  July  1990, 
the  company  was  concentrated  on  design, 
manufacturing  and  marketing  the  BR700  engine  family, 
which  covers  the  12.000  to  22.000  lbs  thrust  range. 
The  BR710,  the  first  variant  of  the  family,  achieved  full 
certification  48  months  after  programme  launch.  The 
BR715,  the  second  variant  of  the  family,  will  be 
certified  in  only  37  months  after  launching  the  project. 
For  future  civil  engines  the  trend  is  likely  to  go  down  to 
a  30  months  product  development  cycle.  To  de-risk  the 
development  programmes,  strategies  must  be  set  in 
place  which  allow  the  introduction  of  advanced 
technologies  with  an  acceptable  level  of  risk. 
Consequently,  advanced  technology  must  be  build  up 
in  separate  research  and  technology  acquisition 
programmes.  The  validation  of  the  most  promising 
technologies  has  to  take  place  in  a  representative 
engine  environment  in  order  to  minimise  the  risk  for 
future  applications. 

BMW  Rolls-Royce  therefore  decided  to  use  the  BR700 
core  engine  as  platform  for  component  validation  and 
certification  tests  at  the  „lnstitut  fur  Flugantriebe"  at  the 
University  of  Stuttgart. 

This  paper  describes  the  set-up  of  the  Stuttgart  ATF  for 
core  testing  and  gives  some  examples  of  how  the 
BR710  and  BR715  development  programmes  have 
been  de-risked  by  using  the  core  vehicle  as  a  mean  of 
technology  readiness  demonstration.  In  addition,  the 
strategy  is  briefly  described  how  to  use  core  testing  as 
a  technology  demonstration  strategy  for  the  21  st 
century. 

2.0  Nomenclature 


Abbreviations 

A/I,  A/0 

Analog  Input/Output 

ARINC 

Aeronautical  Radio  Incorporated 

ATF 

Altitude  Test  Facility 

CMS 

Control  and  Measurement  System 

CPU 

Central  Processing  Unit 

D/I,  D/0 

Discrete  Input/Output 

EEC 

Electronic  Engine  Controller 

EPR 

Engine  Pressure  Ratio 

FADEC 

Full  Authority  Engine  Controller 

FAR 

Fuel  Air  Ratio 

FC 

Facility  Controller 

FMU 

Fuel  Metering  Unit 

HPC 

High  Pressure  Compressor 

HPT 

High  Pressure  Turbine 

ILA 

Institut  fur  Luftfahrtantrlebe 

MC 

Master  Controller 

NOX 

Nitrogen  Oxides 

SOT 

Stator  Outlet  Temperature 

VXI 

VME  Extended  Instrumentation 

Symbols 

NHRT 

Aerodyn.  speed  high-pressure  shaft 

N1 

Mech.  speed  low-pressure  shaft 

N2 

Mech.  speed  high-pressure  shaft 

P 

Total  pressure 

S 

Static  pressure 

SYN 

Synthesised 

T 

Temperature 

t 

Time 

W 

Mass  flow 

WRTQP 

Corrected  mass  flow 

020 

T20/288.15K 

Indices 

20 

Engine  inlet 

25 

Core  engine  inlet 

30 

HP  compressor  exit 

42 

Downstream  valve  42 

57 

Downstream  valve  57 

60 

Upstream  valve  60 

90 

Test  cell 

C 

Command  value 

D 

Demand  value 

M 

Measured  value 

3.0  Introduction 

With  the  formation  of  BMW  Rolls-Royce  in  July  1990, 
the  mission  set  by  the  two  shareholders  BMW  AG  and 
Rolls-Royce  pic.  was  to  become  a  leading  European 
engine  manufacturer.  The  company  has  concentrated, 
from  the  beginning,  on  design,  manufacturing  and 
marketing  a  modem,  highly  competitive  gas  turbine 
engine  family,  which  covers  the  12.000  to  22.000  lbs 
thrust  range.  Ref.  1.  The  BR700  engine  family  is  based 
on  a  common  core  concept  and  different  low  pressure 
system  configurations  to  match  the  aircraft  thrust 
demands  from  the  ultra  long-range  executive  jets  and 
from  the  regional  aircraft  market  up  to  a  size  of  120 
seats.  Figure  1 .  With  the  launch  of  the  engine  core  in 
March  1991  the  BR700  engine  family  was  introduced 
into  a  very  highly  competitive  market  to  provide  a 
family  of  engines  with  best  performance,  lowest  noise 
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level  and  exhaust  emissions,  highest  reiiability  and 
minimum  operational  cost. 

The  BR710,  the  first  variant  of  the  family  was  launched 
In  September  1992  and  has  been  fully  certified  in 
August  1996  for  the  Gulfstream  application,  one  day 
ahead  of  schedule  set  48  months  earlier.  The 
development  time  of  the  BR715  for  the  Boeing  717,  the 
second  variant  of  the  family,  was  already  reduced 
down  to  37  months  from  project  launch  to  certification. 
For  future  civil  engines  the  trend  is  likely  to  go  down  to 
a  30  months  product  development  cycle  to  better 
match  the  aircraft  development  time  scales.  The 
demand  to  develop  competitive  engines  with  the 
highest  avaiiable  standard  of  technology  and  in  parallel 
to  continually  reduce  the  time  from  project  launch  to 
certification,  introduces  more  and  more  risk  to  the 
engine  development  programmes.  Therefore, 
strategies  must  be  set  in  place  to  allow  the  introduction 
of  advanced  technologies  in  the  product  concept  phase 
at  an  acceptable  level  of  risk  for  the  product  realisation 
phase.  Figure  2. 

Within  the  product  realisation  phase,  one  of  the  critical 
milestones  is  always  the  first  run  of  a  new  engine 
followed  by  a  series  of  performance/functional  tests 
and  the  programme  usually  finishes  with  a  number  of 
criticai  certification  tests,  Ref.  2.  In  the  case  of  BR710 
applications  the  flight  test  programme  was  supported 
with  flight  worthy  engines  aiready  in  the  middle  of  the 
BR710  engine  development  programme  before  final 
engine  certification.  In  parallel  to  the  engine  tests  a 
high  number  of  laboratory  and  rig  tests  on  single 
components  and  engine  modules  have  been  carried 
out  in  the  component  development  programme.  Figure 
3.  By  shortening  the  product  realisation  phase  down  to 
30  months,  it  becomes  obvious  that  there  is  more  or 
less  no  time  left  for  the  programme  to  fix  major 
unforeseen  problems.  Consequently,  advanced 
technology  must  be  build  up  in  separate  research  and 
technology  acquisition  programmes  as  shown  in  Figure 
2.  The  BMW  Rolls-Royce  strategy  for  advanced 
technology  acquisition  is  summarised  in  Ref.  3.  One  of 
the  objectives  of  the  programme,  which  is  being 
promoted  by  the  Federal  Ministry  of  Education, 
Science,  Research  and  Technology,  is  to  establish  a 
competitive  technology  base  from  which  core 
engineering  skills  can  be  developed  and  sustained. 

For  the  most  promising  technoiogies  the  final  validation 
will  take  place  In  the  form  of  full  scale  technology 
demonstration.  In  addition  to  the  validation  of  engine 
components  with  test-rigs,  this  demonstration  has  to 
take  place  in  a  representative  engine  environment  in 
order  to  minimise  the  risks  for  future 
applications/products. 

BMW  Rolis-Royce  therefore  decided  to  use  an  already 
existing  BR700  core  demonstrator  engine  as  platform 
for  component  validation  and  certification  tests.  At  the 
Jnstitut  fiir  Flugantriebe"  at  the  University  of  Stuttgart, 
a  suitable  altitude  test  faciiity  was  available  and  could 
be  adapted  to  satisfy  the  specific  requirements  for  core 
engine  testing.  Ref.  4. 


The  main  part  of  the  paper  describes  the  set-up  of  the 
Stuttgart  ATF  for  core  testing  and  gives  some 
exampies  of  how  the  BR710  and  BR715  deveiopment 
programmes  have  been  benefited  respectively  and  de- 
risked  by  using  the  core  vehicle  as  a  means  for 
technoiogy  readiness  demonstration.  In  addition,  a 
strategy  is  briefly  described  how  to  use  core  testing  as 
technology  demonstration  strategy  for  the  2ist  century. 


4.0  Objectives  of  Core  Engine  Testing 
4.1  Core  Engine 

A  ‘core  engine’  comprises  of  substantially  the  high 
pressure  components  of  the  engine,  a  special  intake 
casing  and  a  final  nozzle  with  an  adjustable  area, 
Figure  4.  In  core  engine  testing,  the  mechanical  and 
aerothermal  integration  of  the  high  pressure  system  is 
to  be  replaced  by  a  computer  controlled  system.  This 
Master  Control  System  co-ordinates  the  operation  of 
the  high  pressure  spool  and  the  computer  controlled 
facility,  which  simulates  the  low  pressure  system.  The 
core  engine  can  then  be  operated  under  steady-state 
and  transient  conditions  under  almost  the  same 
conditions  as  in  an  engine  with  a  low  pressure  system. 


4.2  Advantages  of  Core  Engine  Testing 
Compared  with  entire  whole  engine  tests,  core  engine 
testing  additionally  offers  a  number  of  advantages  for 
the  validation  of  the  high  pressure  components  over 
the  entire  operating  range  of  the  engine: 

-  One  significant  advantage  is  the  increased  cost 
effectiveness  compared  with  a  whole  engine  test  in  an 
ATF.  Low  pressure  components  are  not  required  when 
building  the  test  vehicle.  This  leads  to  lower  assembly 
and  engine  hardware  costs.  The  savings  on  installation 
and  testing  alone  amount  to  approximately  30  %. 

-  Without  a  low  pressure  system  it  is  possible  to  vary 
the  mechanical  rotor  speed  and  the  temperatures  in  the 
core  engine  independently.  This  makes  it  easier  to 
carry  out  thermal  paint  tests  and  allows  to  investigate 
the  blade  dynamics  of  the  high  pressure  system  in  an 
engine  environment  at  high  rotor  speeds  but  at  low 
turbine  temperatures. 

-  Due  to  the  variable  nozzle  geometry,  performance 
investigations  on  the  high  pressure  components  can  be 
carried  out  over  a  wide  operating  range  with  no  LP 
system  constraints.  This  supports  the  optimisation  of 
HP  compressor  and  turbine  operating  points  in  the 
engine  operating  environment. 

-  The  direct  measurement  of  the  core  air  mass  flow 
increases  confidence  in  "spool  derived  values".  It 
allows  the  high  pressure  turbine  to  be  validated  in  the 
same  way  as  a  ‘hot’  turbine  rig.  The  operating  points  of 
the  combustion  chamber  can  be  determined  more 
precisely,  especially  during  altitude  relight  tests. 
Exhaust  emission  analysis  also  has  a  tendency  to  give 
better  quality  results. 

-  Since  the  core  inlet  pressure  and  temperature  can  be 
adjusted  independently,  it  is  easier  to  carry  out 
systematic  surveys  on  the  HP  compressor  and  turbine 
Reynolds  Number  effects. 

-  The  better  accessibility  to  the  high  pressure 
components  makes  it  easier  to  install  and  operate 
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sophisticated  instrumentation  such  as  HP  turbine  blade 
tip  clearance  measurement  and  also  allows  X-rays  to 
be  taken  during  engine  running.  Engine  accessories 
can  be  changed  easily. 

4.3  Core  Engine  Tests  In  the  Stuttgart  ATF 
The  objective  of  the  project  was  to  transfer  core  engine 
testing  to  the  Institut  fOr  Luftfahrtantriebe  (ILA)  of  the 
University  of  Stuttgart,  first  of  all  to  support  the  tests 
being  promoted  by  the  German  Aerospace  Technology 
Programme.  This  involved  the  procurement  and 
provision  of  all  necessary  systems,  modification  of  the 
test  facility,  and  commissioning  of  all  control  and 
measurement  systems  and  the  modified  facility  to 
demonstrate  the  following  capabilities; 

-  The  ILA  altitude  test  facility  supports  the  simulation  of 
altitude  flight  conditions  for  a  civil  core  engine,  like  the 
BR700,  over  a  wide  flight  operating  envelope.  Inlet 
pressures  from  5  to  240  kPa,  temperatures  from  210  to 
430  K  and  inlet  mass  flows  up  to  140  kg/s  can  be 
provided  by  the  facility. 

-  Fast  transient  manoeuvres  with  the  core  engine 
require  that  the  test  facility  delivers  repeatable 
pressure  ramps  of  +/-  20  kPa/s,  even  at  low  flows.  After 
appropriate  modification  including  an  automatic  control 
of  the  valves,  which  control  the  air  flow  to  the  core 
engine,  the  required  pressure  ramps  can  be  adjusted 
infinitely  variable. 

-  The  concept  employed  here  for  core  operation  and 
control  of  the  test  facility  provides  the  necessary 
flexibility  to  suit  the  requirements  for  testing  new 
technologies. 

-  The  high  availability  and  reliability  of  the  plant 
combined  with  new  control  and  sub-systems  for  oil  and 
fuel  supply  etc.  reduce  the  duration  and  cost  of  testing. 


5.0  Concept  and  Test  Configuration 
5.1  Overview 

A  Control  and  Measurement  System  (CMS)  based  on  a 
VXI  bus  system  and  three  HP  workstations  takes  over 
the  control  of  the  entire  test  data  acquisition  and 
conditioning  process  as  well  as  the  control  and 
monitoring  of  all  test  activities.  Special  software 
implemented  in  a  shell  called  CENTAURE  is  used  for 
this  purpose.  The  CMS  communicates  with  the  core 
engine  controller  (EEC)  via  the  ARINC  data  bus  and 
with  the  facility  controller  (FC)  via  analogue  I/O  cards. 
By  means  of  position  commands  from  a  virtual  thrust 
lever,  the  software  controls  the  rotor  speed,  N2,  via  the 
EEC  while  also  protecting  the  engine  and  the  test 
facility  from  mal-functions  and  operating  faults  by 
appropriate  logics. 


5.2  Core  Engine  Demonstrator 
The  hardware  of  the  core  engine  consists  of  the 
turbomachinery  components,  the  annular  combustion 
chamber  and  the  secondary  air  system.  They  are 
mostly  the  BR700  engine  standard  parts.  Figure  4.  The 
combustion  air  is  fed  from  the  plenum  chamber  to  the 
engine  via  a  newly-installed  intake  section 
incorporating  a  venturi  nozzle.  Figures  5  and  6.  A 
diffuser  leads  the  air  into  a  special  intermediate  casing 


(swan  necked  duct),  which  is  derived  from  the  standard 
engine  and  serves  as  the  core  intake. 

The  intermediate  casing  also  provides  the  suspension 
for  the  core  engine  and  is  the  point  where  the  thrust 
forces  are  introduced  into  the  thrust  frame.  The  latter 
also  accommodates  the  fuel  system.  A  water-cooled 
exhaust  unit,  with  a  nozzle  area  that  can  be  adjusted  in 
several  stages,  replaces  the  low  pressure  turbine.  The 
accessories  and  the  control  system  components  such 
as  the  fuel  system,  the  compressor  variable  stator  vane 
mechanism,  the  sensors,  the  ignition  system  and  the 
handling  bleed  valves  are  also  standard  engine 
components. 


5.3  Control  and  Measurement  System 
The  hardware  consists  of  modular  design  and 
essentially  comprises  a  VXI  bus  system  with  two 
chassis  and  a  total  of  25  plug-in  modules.  Three  of 
these  25  cards  are  CPU’s  (HP  743i,  100  MHz).  Two 
CPU’s  execute  the  test  data  acquisition.  The  third  CPU 
runs  the  software  for  overall  control  and  is  hence  called 
the  Master  Controller  (MC).  In  addition,  there  are  five 
A/I,  four  A/O,  one  D/0,  two  D/I  cards  and  one  timer 
card  installed.  One  MXI  card  per  chassis  is  available 
for  inter-chassis  communication.  One  ARINC  card 
handles  the  communication  between  the  CMS  and  the 
EEC. 

The  measurement  system  supports  more  than  500 
thermocouples  of  various  types,  1000  pressure  signals 
provided  by  ILA’s  Scanivalve  System,  and  500  M’s,  48 
A/O’s,  128  D/O’s  and  64  D/I  channels.  All 
measurement  channels  except  the  rotating  Scanivalve 
System  can  also  be  used  as  input  parameters  for 
control  purposes. 

A  software  package  provides  open-loop  and  closed- 
loop  control  for  all  test  activities.  It  is  a  real-time 
programme  and  has  been  designed  for  both  manual 
and  automatic  operation  of  the  integrated  control 
system.  In  the  automatic  operation  or  closed-loop 
control  mode  it  is  possibie  to  employ  four  different 
operation  modes;  Windmilling,  Starting,  Slow  Transient 
and  Fast  Transient. 

Special  modes  in  the  CMS  provide  the  requested  inlet 
conditions  for  the  core  engine  dependent  on  the 
requirements  of  the  test.  The  core  engine  inlet  and 
outlet  conditions  can  be  commanded  directly,  e.g.  as 
schedules  versus  the  N2  speed  or  exclusively  through 
altitude  flight  conditions.  This  allows  to  simulate  the 
performance  characteristics  of  the  low-pressure 
compressor  correctly. 

The  aerothermal  and  mechanical  constraints  on  the 
operating  range  of  the  core  engine  by  the  low  pressure 
system,  are  not  present.  Certain  limiters  on  the 
principal  process  parameters  are  implemented  in  the 
MC  software  in  addition  to  several  EEC  parameters. 
They  provide  protection  for  the  whole  system  against 
excessive  operating  conditions. 
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5.4  Core  Engine  Controller 

A  FADEC  similar  to  that  used  on  the  BR700-710 
development  engines  for  the  Gulfstream  V,  was  chosen 
as  core  engine  controller  and  appropriately  modified. 
The  EEC  as  the  main  component  of  the  FADEC  is  an 
electronic  control  system  with  two  independent  lanes. 
Each  one  has  its  own  CPU  and  power  supply.  Using 
commands  and  information  supplied  by  the  ARINC  429 
Digital  Data  Bus  and  from  various  sensors  as  input 
signals,  the  EEC  regulates  the  engine  thrust  through  a 
closed-loop  control  by  varying  the  position  of  the  fuel 
metering  valve  in  the  FMU. 

In  the  primary  mode  the  EEC  uses  EPR  control  and  in 
the  ‘reversionary  mode’  it  uses  N1  control.  The  core 
engine  provides  no  speed  signal  from  a  low-pressure 
shaft  (N1).  Therefore  the  control  laws  were  altered  in 
order  to  implement  N2  as  the  controlled  variable.  The 
N1  control  loop  is  modified  to  a  continuously  active  N2 
control  loop.  The  EPR  control  loop  is  completely 
deactivated.  The  input  signal  for  the  thrust  lever 
position  command  is  modified  to  a  N2  command  that  is 
no  longer  transmitted  as  an  analogue  signal,  but  now 
for  the  first  time  in  engine  testing  via  the  ARINC  data 
bus  as  digital  signal. 

The  EEC  software  was  adapted  to  implement  the  new 
control  laws.  They  contain  a  primary  mode  that 
provides  closed  loop  control  of  the  mechanical  shaft 
speed  over  the  whole  flight  operating  envelope  of  the 
engine. 


5,5  Test  Facility  and  Facility  Controller 
The  facility  operation  diagram  shows  the  most 
important  part  of  the  altitude  test  facility  for  running  the 
core  demonstrator  tests.  Figure  7.  The  inlet  air  of  the 
core  engine  is  supplied  by  compressor  VO  and 
compressed  to  meet  the  engine  intake  pressure 
requirements.  During  altitude  testing  with  sufficient  low 
engine  inlet  pressures,  the  air  can  also  be  taken 
directly  from  the  facility  environment  by  using  a  suction 
compressor  on  the  engine’s  exhaust  side.  The 
adjustment  of  the  core  engine  inlet  parameters  is 
carried  out  by  the  valves  L42  and  L57  in  the  two 
parallel  ducts. 

The  bypass  valve  LOO  in  parallel  with  the  core  engine 
diverts  air  round  the  core  engine  during  rapid 
decelerations  or  under  part-load  conditions,  thereby 
reducing  the  dead  time  for  passing  through  the  inlet 
section. 

During  fast  transients,  where  rapid  changes  in  pressure 
and  temperature  at  the  inlet  of  the  core  engine  are 
required,  the  volume  between  the  valves  L42  and  L57 
and  the  core  engine  is  critical  for  the  transient 
behaviour  of  the  facility.  For  a  slam  acceleration  and  a 
resulting  rise  in  the  inlet  pressure  P25,  this  volume  has 
to  be  filled  with  air.  A  mass  flow  of  air  several  times 
greater  than  the  maximum  engine  flow  capacity  has  to 
be  supplied  via  the  valves  L42  and  L57.  When 
adjusting  the  inlet  temperature  T25  the  air  takes  time  to 
flow  through  that  volume  and  hence  produces  a  dead 
time.  The  heat  exchange  with  the  structural  parts 


causes  a  change  in  temperature  and  has  to  be 
considered. 

To  meet  these  requirements  the  FC  performs  a  closed- 
loop  control  of  the  valves  L42,  L57  and  L60.  The  MC 
provides  the  FC  with  the  command  values  P25_C  and 
T25_C,  with  the  core  engine  flow  demand  W25_C  and 
the  rate  of  change  of  pressure  dP25/dt_C.  The 
temperature  T25  is  open-loop  controlled  by  varying  the 
mixing  temperature  according  to  the  command  value 
T25_C  issued  by  the  MC.  In  addition  the  heat 
exchange  with  the  structure  is  taken  into  account  by  an 
appropriate  model.  The  closed-loop  pressure  control 
used  for  correcting  the  open-loop  control  of  P25  acts 
by  means  of  a  proportional  element  and  an  integral 
element  by  varying  the  command  value  of  dP25/dt_C. 
For  temperature  control,  the  desired  value  of  the  mixing 
temperature  is  varied  by  an  integral  element. 

A  166  MHz  PC  was  used  as  FC  for  processing  the 
measured  values  and  the  data  supplied  by  the  MC, 
calculating  the  open-loop  and  closed-loop  control 
operations,  and  providing  the  valve  control  signals. 


5.6  Sub-Systems 

Unlike  the  entire  whole  engine,  the  BR700  core  engine 
does  not  have  a  bevel  gear  drive  from  the  high 
pressure  shaft  that  powers  the  flight  engine  gearbox, 
hence  it  is  necessary  to  incorporate  a  number  of  sub¬ 
systems. 

-  A  self-contained  N2-controlled  lubricating  oil  system 
carries  out  the  oil  supply,  vent  and  scavenge  functions 
for  the  two  bearing  chambers  of  the  core  engine.  The 
system  supports  -  by  means  of  a  precise  measurement 
of  the  bearing  oil  and  air  pressures  and  temperatures  - 
the  investigation  of  oil  balance  or  new  sealing 
technologies. 

-  The  fuel  system  uses  the  BR710  standard  fuel  pump, 
fuel  metering  unit  (FMU)  and  fuel-cooled  oil  cooler.  An 
electric  motor  with  variable  speed  control  provides  the 
power  to  the  high  pressure  fuel  pump. 

-  A  closed  loop  cooling  water  system  provides 
continuous  circulation  of  pressurised  water  to  the 
exhaust  unit.  Various  cooling  air  systems  provide  for 
example  cooling  of  the  high-pressure  turbine  casing  to 
maintain  a  defined  value  for  blade  tip  clearances. 

-  To  simulate  the  aircraft  cabin  air  and  anti-icing 
system,  bleed  air  is  taken  off  the  core  engine  through 
butterfly  valves  which  are  driven  by  a  self-contained 
hydraulic  unit  and  controlled  by  the  MC. 


6.0  Commissioning  and  Validation 

6.1  Interaction  of  the  Systems 

The  interaction  between  various  components  of  the 
system  is  illustrated  in  Figure  8.  Before  the  test  starts, 
the  control  operator  of  each  of  the  control  systems 
needs  to  provide  the  inputs  for  his  system  which  is 
required  for  co-ordinated  operation..  All  sub-systems 
are  configured  prior  to  test. 

During  testing,  the  operator  feeds  the  demand  values 
for  speed  N2  and  the  required  flight  conditions  into  the 
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MC.  The  MC  generates  the  corresponding  N2 
command  values  in  order  to  arrive  at  the  target  value  of 
N2  in  incremental  stages.  The  command  values  are 
passed  on  both  to  the  sub-systems  and  to  the 
synthesis  tables.  The  synthesis  tables  produce  the 
command  values  for  the  core  engine  inlet  and  outlet 
conditions  and  a  synthesised  EPR  command  to  feed 
the  EEC  functions  dependent  on  EPR.  The  N2  and  the 
EPR  command  values  are  sent  to  the  EEC.  Control  is 
exercised  by  the  closed-loop  N2  control  circuit  of  the 
EEC. 

The  command  values  for  the  core  engine  Inlet 
conditions  (P25_C,  T25_C,  W25_C,  dP25/dt_C)  are 
sent  to  the  PC.  The  PC  provides  open-loop  and  closed- 
loop  control  of  the  core  engine  inlet  pressure  as  well  as 
adjusting  the  inlet  temperature  to  the  set  value  in  a 
closed  control  loop. 

The  whole  system  can  be  operated  under  automatic 
control  over  a  wide  operating  range,  and  very  fast  rates 
of  change  of  N2,  P25  and  T25  can  be  achieved. 


6.2  Integrating  the  Systems 

A  series  of  tests  was  performed  on  a 
Hardware/Software  Integration  Rig  in  order  to  integrate 
the  EEC  and  the  MC.  The  testing  verified  the 
communication  between  the  systems,  the  stability  and 
various  emergency  functions  over  a  defined  range  of 
the  engine  flight  operating  envelope. 

The  ATP  and  the  PC  were  used  to  carry  out  special 
preliminary  tests  with  a  throttling  valve  in  place  of  the 
core  engine  for  simulation  purpose.  This  was  to 
demonstrate  how  good  the  required  inlet  conditions  for 
the  planned  transient  core  engine  tests  on  the  ATP  can 
be  achieved  by  the  computer  controlled  facility. 

The  installation  of  the  systems  on  the  test  facility  in 
Stuttgart  was  finalised  by  an  acceptance  test 
procedure  concentrating  on  the  communication  first. 
Then,  the  operating  range  of  the  engine  was  expanded 
from  idle  speed  to  maximum  power  and  the  functions  of 
the  control  systems  were  verified  successfully.  Slow 
transient  testing  continued  with  the  inlet  pressure  and 
temperature  increasing  with  the  speed.  The  behaviour 
of  the  pressure  to  follow  the  command,  in  particular, 
proved  to  be  excellent.  Pinally  the  acceleration  and 
deceleration  times  were  subsequently  reduced  further 
to  slam  manoeuvres. 

Accelerations  from  high  idle  speed  to  maximum  power 
were  achieved  in  approximately  4  seconds  at  sea  level 
static  conditions  and  approximately  10  seconds  at 
40,000  ft  altitude,  Pigure  9.  Since  the  dead  time  in  the 
temperature  control  could  only  partly  be  compensated 
by  the  compression  or  expansion  effect,  notable 
divergences  occur  in  the  actual  temperature  settings. 
In  future,  however,  it  is  possible  to  compensate  for 
most  of  the  slight  lag  of  the  temperature  during  the 
rapid  acceleration  by  installing  an  extra  dead  time 
element  to  allow  the  temperature  signal  to  be  issued 
with  a  lead  time. 


6.3  Core  Engine  Tests 

During  and  after  commissioning  a  series  of 
experiments  to  support  the  BR715  Engine 
Development  Programme  and  the  German  Aerospace 
Research  Programme  were  carried  out  with  BR710  and 
BR715  standard  core  engine  components. 

6.3.1  Emission  Gas  Measurement.  An  emission  gas 
analysis  test  was  carried  out  to  provide  the  datum 
emission  levels  of  the  core  engine  with  a  standard 
combustor.  This  was  necessary  for  comparison  with 
future  emission  test  results  gained  from  a  core  with  a 
staged  combustor. 

Compared  to  combustor  rigs,  the  emission  gas  analysis 
on  a  core  engine  allows  to  verify  the  emission  levels  of 
a  full  annular  combustor  in  an  engine  environment  with 
representative  pressure  and  temperature  levels.  Due  to 
the  gas  sampling  directly  in  the  core  gas  path,  the 
measurement  in  a  core  engine  provides  higher  quality 
results. 

Additionally  the  core  engine  and  facility  will  be  matched 
aerothermally  to  simulate  various  sea  level  and  altitude 
flight  conditions  for  the  whole  range  of  derivatives  of 
the  BR700  engine  family.  This  is  possible  by  adjusting 
the  aerothermal  combustor  inlet  conditions  at  a 
constant  fuel-air-ratio.  Figure  10. 

A  mobile  emission  gas  analysis  system  controls  the 
whole  measurement  process  of  calibration,  data 
recording  and  analysis.  According  to  ICAO  standard 
the  system  uses  the  following  methods  for  gas 
analysis: 

-  SAE  smoke  meter  (filter  technique)  for  smoke 
detection. 

-  Non-dispersive  infra-red  analyser  (NDIR)  for  CO  and 
C02  detection. 

-  Flame  ionisation  detector  (FID)  for  UHC  detection. 

-  Chemi-lumniscent  detector  (CLD  method)  for  NOX 
detection. 

Additionally,  to  determine  the  combustion  efficiency, 
the  concentration  of  Oxigen  and  Nitrogen  is  being 
measured  as  well  as  the  air  humidity  which  influences 
the  measuring  method  based  on  infra-red  analyser. 

The  tests  confirmed  the  excellent  emission  standard  of 
the  BR700  combustor.  The  measured  value  of  the 
BR710  NOX  emission  level  provides  a  margin  of  45  % 
to  the  current  ICAO  limit.  Compared  to  other  engine 
types  the  BR710  with  the  standard  combustor  is 
already  one  of  the  leading  low  emission  engines. 

6.3.2  Thermal  Paint  Test.  A  thermal  paint  test  with 
BR715  standard  combustor  and  high  pressure  turbine 
was  carried  out  on  the  core  engine  to  support  the 
BR715  development  programme.  This  test  was  used  to 
investigate  and  evaluate  the  surface  temperature 
distribution  of  the  hot  end  components.  The 
components  of  high  interest  were  the  stage  one  and 
two  turbine  blades  and  the  combustor  can  as  well  as 
heat  shields  and  burners.  Furthermore  the  temperature 
distribution  of  the  HP  turbine  discs  and  the  combustor 
and  turbine  casings  were  assessed.  By  means  of  using 
various  types  of  turbine  blades  with  six  different 
thermal  paint  types  and  also  different  blade  cooling 
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design  standards  (with  different  cooling  hole 
configurations),  it  was  possible  to  assess  the  efficiency 
of  the  turbine  blade  cooling  system,  Figure  11. 

Although  this  test  was  a  very  short  one,  it  is  also  one  of 
the  most  difficult  tests  as  it  allows  for  only  one  attempt 
once  the  core  is  lit.  The  core  engine  and  the  inlet 
conditions  had  to  be  matched  aerothermally  to  achieve 
the  required  target  conditions  (HPC  exit  and  HPT  entry 
temperatures).  This  is  very  important  for  accurate 
results.  The  stabilisation  time  of  about  5  minutes  on 
condition  with  maximum  stator  outlet  temperature 
ensured  that  the  thermal  paint  reached  a  stabilised 
temperature  with  minimum  thermal  paint  loss  due  to 
erosion  effects.  An  overshoot  of  the  inlet  conditions 
would  have  changed  the  thermal  paint  colours  to  a  less 
accurate  temperature,  with  no  opportunity  for  corrective 
actions,  causing  this  part  test  to  have  failed.  The 
accurate  core  engine  control  guaranteed  the  right  inlet 
conditions  and  the  test  objective  was  achieved. 

Based  on  the  validated  test  data,  a  redesign  of  the 
turbine  blade  cooling  system  was  incorporated  during  a 
relatively  early  design  stage.  The  thermal  behaviour  of 
fuel  injectors  and  combustion  heat  shields  with  blocked 
cooling  holes  was  also  assessed.  Furthermore  the 
validation  of  the  aerothermal  models  was  possible  by 
comparison  of  the  acquired  test  data  against  the 
calculated  values. 

6.3.3  Vibration  Survey  Test.  Vibration  surveys  of  the 
modified  HPC  blading  and  of  the  new  HPT  blading 
were  carried  out  to  support  the  BR715  development 
and  certification.  Especially  the  HPT  certification  test 
covered  the  whole  operating  range  of  the  engine  incl. 
an  overspeed  margin  of  5  %.  The  target  is  to  prove  that 
the  Eigenfrequencies  and  stresses  of  the  blades  are 
not  critical  to  the  integrity  of  the  turbine. 

During  the  slow  accel  to  a  maximum  mechanical  rotor 
speed  which  is  5  %  above  the  value  the  BR715  engine 
will  ever  see  in  service  and  during  the  following  decel 
the  stress  signals  were  measured  by  the  strain  gauges 
on  the  blades  and  transmitted  via  a  slip  ring  unit  to  the 
mobile  integrated  dynamic  data  analysis  system.  To  ' 
keep  the  turbine  temperatures  low,  the  test  was  carried 
put  with  an  increased  final  nozzle  area  and  more 
closed  variable  guide  vanes  of  the  HPC.  As  during  this 
test  very  high  levels  of  HPC  exit  and  HPT  entry 
temperatures  are  achieved,  it  is  very  important,  that  the 
facility  delivers  a  steady  core  inlet  temperature  to  avoid 
turbine  overtemperature.  Vibration  surveys  of  the 
turbine  have  to  be  successful  from  the  beginning,  as 
the  life  time  of  the  strain  gauges  on  the  hot  biades  is 
very  limited. 

Especially  a  core  engine  test  supports  the  combination 
of  a  vibration  survey  and  thermal  paint  test  during  one 
test  campaign. 


7.0  Prospects  for  Future  Testing 

7.1  Staged  Combustor  Core  Demonstrator 
The  next  test  campaign  concentrates  on  the  validation 
of  a  core  demonstrator  equipped  with  a  fuel  staged, 
low-emission  dual  annular  combustion  chamber.  There 
will  be  several  technical  challenges  employed  by  the 
new  technology. 


The  fuel  staged  combustor  consists  of  a  pilot  fuel 
injector  stage  for  low  power  and  an  additional  main  fuel 
injector  stage  for  high  power  running,  Figure  12.  The 
increased  complexity  of  the  systems  will  cause  an 
additional  challenge.  A  staging  valve  control  unit  has 
been  developed  to  execute  specified  controi  algorithms 
and  to  provide  closed-loop  control  of  the  fuel  staging 
valve  position.  All  additional  functions  wili  be  integrated 
in  the  existing  hardware  and  software  of  the  systems. 
The  special  challenge  of  this  system  will  be  to  exactly 
control  the  split  of  the  fuel  between  the  two  injector 
stages,  especially  during  transients.  This  will  guarantee 
high  efficiency,  high  safety  against  flame  out,  low 
emission  and  fast  response  during  engine  transients. 
The  main  targets  will  be  to  demonstrate  a  reduction  in 
NOX  emission  to  a  level  of  35  %  of  the  current  ICAO 
limit  and  to  compare  the  handling  characteristics  of  the 
engine  with  a  conventional  combustor  engine.  Figure 
13.  Testing  will  cover  the  whole  flight  operation 
envelope  and  will  include  relight  testing  and  critical 
handling  cases. 

A  second  test  campaign  will  be  used  to  additionally 
validate  advanced  technology  which  is  currently  in  its 
development  phase: 

-  The  staged  combustor  will  undergo  a  second  test  to 
validate  an  improved  standard  of  the  can  and  the  fuel 
and  control  system. 

-  Blisk  technology  in  the  HP  Compressor  and  a 
composite  hybrid  variable  stator  vane  mechanism  will 
help  to  reduce  engine  weight. 

-  An  innovative  heat  management  system  and  seal 
design  for  the  HP  turbine  will  improve  the  specific  fuel 
consumption  by  decreasing  the  amount  of  cooling  air 
for  the  HPT,  it  will  increase  the  life  of  the  turbine  discs 
and  will  reduce  weight. 

The  testing  again  will  concentrate  on  the  demonstration 
of  the  reduced  level  of  emission  and  will  be  used  to 
optimise  the  engine's  handling  capabilities  over  the 
whole  flight  operation  envelope.  Surge  tests  and  water 
ingestion  tests  will  become  necessary  to  evaluate  the 
response  of  the  new  combustor  and  control  system  at 
unforeseen  events  and  Inclement  weather  conditions 
during  operation. 


7.2  Strategy  for  Future  Core  Engine  Technology 
Due  to  the  increasing  air  traffic  the  main  technical 
targets  for  future  engine  concepts  are  to  decrease  fuel 
consumption,  emission  and  noise.  The  commercial 
challenges  will  be  to  reduce  development,  production 
and  maintenance  cost  and  to  increase  the  reliability  of 
the  engine.  To  fulfil  these  requirements  and  to  remain 
competitive,  new  technology  has  to  be  acquired. 

Within  the  initiative  of  the  German  Aerospace 
Research  Programme  Part  "Engine  3E"  (Efficiency, 
Economy,  Environment),  which  is  being  promoted  by 
the  Federal  Ministry  of  Education,  Science,  Research 
and  Technology,  a  technological  base  for  future 
competitive  engine  development  skills  will  be 
established. 

The  foliowing  targets  for  an  engine  concept  with  a 
bypass  ratio  beyond  12  and  an  overall  pressure  ratio  of 
50  will  be  aimed  at  Figure  14: 
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The  German  Aerospace  Research  Programme  is  going 
to  enter  the  second  concept  phase,  it  wiil  comprise  the 
development  of  an  advanced  technology  core 
demonstrator  with  new  compressor,  combustor  and 
turbine  components  and  will  undergo  validation  testing 
in  about  3  years  time.  This  demonstrator  will  be 
characterised  by  a  high  pressure  ratio  HPC  (>  22)  with 
increased  exit  temperature  and  3D-Wide-Chord- 
Blading,  an  extremely  low-emission,  dual  annular 
combustor  with  a  staged  fuel  system,  a  2-stage  high 
temperature  HPT  with  an  optimised  cooling  concept  and 
a  HPT  Stator  Outlet  Temperature  beyond  1800  K  and 
an  advanced  control  system. 

A  wide  range  of  tests  supporting  a  future  certification 
programme  will  be  planned  and  used  to  validate  the 
new  technologies. 

-  Vibration  surveys  for  the  investigation  of  the  blade 
dynamics  using  telemetry  to  transmit  the  strain  gauge 
signals. 

-  Performance  investigations  over  the  whole  flight 
operation  envelope  to  verify  the  benefit  in  fuel 
consumption. 

-  Component  performance  investigation  will  comprise 
tests  to  map  the  compressor  and  turbine  characteristics 
and  to  verify  Reynolds  No.  effects. 

-  Emission  gas  measurement  and  analysis  to 
demonstrate  the  reduction  in  NOX  of  the  new 
combustor  standard. 

-  Handling  tests  will  be  used  to  optimise  the  response  of 
the  staging  valve  control  logic. 

-  Simulated  inclement  weather  condition  testing  will 
verify  the  combustor's  robustness  against  flame  out  and 
provide  data  to  set  up  a  control  logic  for  inclement 
weather  condition. 

-  Thermal  survey  testing  to  validate  the  thermal  model 
by  the  use  of  rotating  temperature  measurements  and 
by  running  square  cycles. 

-  Blade  tip  clearance  measurements  will  be  used  to 
validate  the  running  clearances  of  the  turbomachinery 
components. 

-  Thermal  paint  testing  to  verify  the  surface 
temperatures  of  the  new  blading  and  other  components. 

-  Endurance  testing  for  cyclic  evaluation  of  the  hot  end 
components  will  require  fast  transients  with  a  schedule 
of  inlet  pressure  versus  rotor  speed  and  a  constant  inlet 
temperature. 

With  the  advanced  core  concept  the  BR700  engine  has 
the  potential  for  35000  lbs  thrust.  A  possible  time  scale 
for  development  and  certification  is  shown  in  Figure  15. 


8.0  Conclusions 


technology  in  each  new  product,  adds  more  and  more 
risk  to  the  product  realisation  phase.  A  strategy  has 
been  described  how  to  introduce  new  technology  in  the 
product  development  phase  with  an  acceptable  level  of 
risk.  As  a  consequence,  BMW  Rolls-Royce  has 
selected  the  BR700  core  engine  as  platform  for 
technology  validation  using  the  Stuttgart  ATF. 

A  series  of  testing  has  demonstrated,  that  core  testing 
has  been  successfully  used  to  support  the  BR710  and 
BR715  certification  programme.  Therefore,  our  ability  to 
run  core  engines  on  the  ATF  in  Stuttgart  has 
established  a  solid  base  on  which  future  technological 
projects  can  be  developed.  More  testing  are  planned  for 
1998/99  and,  in  addition  to  the  validation  of  the  staged 
combustor,  tests  are  planned  for  the  validation  of  a 
BLISK  HP  compressor  with  composite  hybrid  variable 
stator  vanes  and  an  innovative  heat  management 
system  and  seal  design  for  the  HP  turbine. 

Our  mid/long  term  plans  for  the  second  phase  of  the 
German  Aerospace  Research  Programme 
demonstrates,  that  core  engine  testing  is  an  integral 
part  of  our  strategy  for  the  21  st  century. 
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10.0  Figures 


Figure  1 : 

BR700  Engine  Family  Concept 
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Figure  2: 

Strategy  how  to  introduce  advanced  technologies  into  future  products 
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Figure  3: 

Schematic  of  a  typical  Engine  Development  Programme 


Figure  4: 

Cross-section  of  BR700  Core  Engine  Demonstrator 
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Figure  5: 

General  Arrangement  of  the  Test  Bed 


Figure  6: 

Test  Cell  with  Core  Demonstrator 
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Figure  8: 

Control  Systems  Integration 
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Figure  9: 

Commanded  and  Measured  Values  of  the  Core  Engine 
Inlet  Parameters  at  Sea-Level  Static  Conditions 


Figure  10: 

Simulation  of  various  BR700  Engine  Types  for  Emission  Gas  Analysis  Testing 


12.12 
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Fuel  Staged  Dual  Annular  Combustor 


12.13 


Figure  13: 

Potential  for  NOX  Production  of  New  Combustor  Technology 


Figure  14: 

Example  of  an  Engine  Concept  within  the  German  Aerospace  Research  Programme 


Figure  15: 

Concept  of  the  planned  second  phase  of  the  German  E3E  Technology  Programme 
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Abstract 

The  United  States  Air  Force  (USAF)  developed  a  new 
vision  {Global  Engagement:  A  Vision  for  the  21^1 
Century  Air  Force  [USAF  1997])  to  exploit  the 
strengths  of  modem  air  and  space  power  through  a  set 
of  core  competencies,  guide  the  evolution  of  the  USAF 
from  an  “Air  and  Space  Force”  to  a  “Space  and  Air 
Force,”  and  provide  a  focus  for  21"  century  USAF  Test 
and  Evaluation  (T&E).  Future  implementation  of  this 
vision  depends  on  the  pursuit  of  a  number  of 
technologies  -  efforts  that  will  have  an  impact  on  both 
required  T&E  capabilities  and  T&E  technology  needs. 

This  paper  addresses  ongoing  efforts  to  meet  these  goals 
by  discussing  the  test  technology  needs  identification 
process  trail  from  visionary-type  documents  such  as 
Joint  Vision  20 JO,  America's  Military:  Preparing 
for  Tomorrow  (J.  Shalikashvili  1995)  and  Global 
Engagement:  A  Vision  for  the  21^^  Century  Air  Force 
(USAF  1997),  to  T&E  capability  requirements 
definition,  and  to  the  identification  of  future  test 
technology  needs.  This  paper  specifically  addresses  the 
efforts  of  the  USAF  T&E  communities’  development 
of:  (1)  a  USAF  T&E  Mission  Support  Plan  (MSP),  (2)  a 
Tri-Service  Required  T&E  Capabilities  Analysis 
(RTECA),  (3)  USAF  Annual  Planning  and 
Programming  Guidance  (APPG),  and  (4)  a  USAF  T&E 
Requirements  Assessment  Team  (TERAT)  database.  A 
new  notional,  formalized  process  for  test  technology 
needs  identification  using  these  studies  and  plans  as  a 
basis  will  be  presented.  This  new  process  provides  for 
integrating  existing  and  future  warfighter  requirements 
and  strategic  guidance  into  identifiable  T&E  capability 
requirements,  which  in  turn,  allows  for  identification 
and  discussion  of  new  test  technology  needs  for  21" 
century  USAF  T&E. 

Introduction 

United  States  Air  Force  T&E  is  an  outgrovrth  of  a 
rich  weapons  systems  research,  development,  and 
acquisition  heritage.  Test  and  evaluation  evolved  to 
support  the  development  of  weapons  systems  necessary 
to  protect  our  national  interests.  Preparing  for  the  future 
of  T&E,  especially  in  today’s  uncertain  post-Cold  War 
environment,  presents  a  multitude  of  new  and  difficult 


challenges.  What  we  will  have  to  test  and  how  we  will 
test  are  changing.  Other  factors  such  as  consolidation, 
environmental  issues,  acquisition  reform,  and  budget 
reductions  will  require  changes  in  our  organizations, 
processes,  and  infrastructure.  Our  focus  must  be  clear 
and  our  game  plan  coordinated;  we  must  improve 
efficiency  and  reduce  overhead  while  preparing  the 
people  and  infrastructure  to  meet  the  T&E  demands  of 
future  weapon  systems.  One  of  the  ways  to  do  this  is  to 
improve  the  technology  of  our  T&E  systems. 

However,  numerous  questions  exist  regarding  the  future 
of  USAF  T&E.  What  essential  capabilities  must  be 
present  in  the  T&E  infrastructure  between  now  and  the 
year  2010?  What  capabilities  are  required  beyond  2010? 
What  improvements  are  required?  Where  do  we  start  the 
process  of  test  technology  needs  identification  for  the 
USAF  commimity?  What  provides  the  guidance  for 
directing  us  into  die  21"  centuiy? 

The  USAF  has  been  working  diligently  for  the  past 
several  years  to  answer  these  questions  and  to  prepare 
for  the  21"  century.  A  view  of  die  test  technology  needs 
identification  trail  from  top-level  vision  to  needs 
identification  will  be  presented.’  We  will  only  touch 
briefly  on  the  vision  and  guidance,  and  concentrate  on 
four  studies  that  the  USAF  has  either  chartered  or 
participated  in.  We  will  outline  the  objective,  scope  of 
the  effort,  process  used,  limitations,  and  present  a 
summary  of  each  of  the  studies.  The  intention  is  to  show 
there  currently  is  no  single  process  for  test  technology 
needs  identification  in  the  USAF.  We  will  illuminate  the 
positive  aspects  and  discuss  the  limitations  of  each  of 
the  studies,  and  attempt  to  provide  a  new  notional, 
singular  process  for  identification  of  test  technology 
needs  for  the  21"  centuiy  USAF. 

Visions 

We  must  start  at  the  top  in  the  test  technology  needs 
identification  process.  Many  of  the  visionary  type 
dociunents  are  very  abstract,  and  thus  necessarily  macro 
in  nature  and  contain  very  few  references  to  specific 


’One  area  tiiat  requires  clarification  to  avoid  confusion  is  the  use  of  die  tenns 
‘T&E  Requiremenf’  versus  ‘T&E  Need.”  A  ‘T&E  Requirement”  is  best 
described  as  “what  T&E  tasks  we  need  to  accomplish.”  A  ‘T&E  Need”  is 
described  as  “what  assets  we  need  to  conduct  a  particular  type  of  test” 


weapon  systems,  which  makes  it  difficult  to  link  to 
T&E.  However,  these  documents  constitute  the  starting 
point  for  our  modernization  planning  process.  The 
USAF  T&E  community  looked  to  the  Joint  Chiefs  of 
Staff  for  this  start.  General  John  M.  Shalikashvili,  then 
Chairman  of  the  Joint  Chiefs  of  Staff  provided  the 
conceptual  template  for  how  America’s  Armed  Forces 
will  fight  in  the  early  21"  century.  This  template  is  Joint 
Vision  2010,  America’s  Military:  Preparing  for 
Tomorrow  (Reference  1).  JV2010,  as  it  is  called, 
provides  guidance  on  how  we  will  leverage 
technological  opportunities  to  achieve  new  levels  of 
effectiveness  in  joint  warfighting. 

Joint  Vision  2010 

The  Chairman  of  the  Joint  Chiefs  of  Staff  Joint  Vision  2010 
(JV2010)  gives  us  some  indication  of  how  the  United  States 
(U.S.)  expects  warfi^ting  to  be  conducted  in  the  future, 
and  thus  what  general  types  of  technology  the  USAF  T&E 
community  must  be  prepared  to  evaluate.  This  vision  of 
future  warfighting,  assuming  improved  intelligence  and 
command  and  control  to  be  available  in  the  “information 
age,”  goes  on  to  develop  four  operational  concepts: 
dominant  maneuver,  precision  engagement,  fiill 
dimensional  protection,  and  focused  logistics.  JV2010 
proclaims  that  “full  spectrum  dominance  will  be  the  key 
characteristic  we  seek  for  our  Armed  Forces  in  die  21" 
century.”  These  four  concepts  help  to  achieve  that 
dominance.  JV2010  further  proclaims  “Imperative  of 
jointness,  we  must  find  the  most  effective  methods  for 
integrating  and  improving  interoperability;  we  must  be  fully 
joint;  institutionally,  organizationally,  intellectually,  and 
technically.”  We  must  attempt  to  incorporate  jointness  into 
each  of  the  four  operational  concepts. 

JV2010  states  “Major  improvements  in  U.S.  military 
capabilities  should  arise  fi-om  simultaneous 
developments  in  battlespace  awareness;  advanced 
command,  control,  communications,  computing,  and 
intelligence  (C4I);  and  precision  force.  Individually, 
these  new  systems  portend  sharply  increased 
effectiveness.  Collectively,  they  promise  to  widen  the 
lead  of  the  U.S.  military  over  its  competitors,  even  in 
the  face  of  declining  defense  budgets.”  Called  the 
“system  of  systems”  military  instrument  by  then  Vice 
Chairman  of  the  Joint  Chiefs  of  Staff  Admiral  Owens, 
the  T&E  community  can  expect  to  be  asked  to  evaluate 
multi-platform  weapon  systems  or  systems  of  weapon 
systems  in  the  future. 

JV2010  identified  several  advancing  technology  trends. 
Long-range  precision  capability,  combined  with  a  wide 
range  of  delivery  systems,  is  emerging  as  a  key  factor  in 
future  warfare.  The  ability  to  produce  a  broader  range  of 
potential  weapons  effects  will  enhance  precision 
capability.  Advances  in  low  observable  technologies 


and  the  ability  to  mask  fnendly  forces  will  continue. 
Improvements  in  information  and  systems  integration 
technologies  will  significantly  impact  future  military 
operations  by  providing  decision  makers  with  accurate 
information  in  a  timely  manner,  gaining  them  dominant 
battlespace  awareness  for  the  U.S. 

JV2010  concludes  “The  combination  of  these  technology 
trends  will  provide  an  order  of  magnitude  improvement  in 
lethality.”  The  “increasingly  lethal  battlespace  will  increase 
the  importance  of  stealth,  mobility,  dispersion,  and  the 
ability  to  operate  at  a  higher  tempo.”  “Greater  mobility  and 
increased  dispersion  will,  in  turn,  require  additional 
communications  and  coordination  capabilities  since  the 
synchronization  of  these  dispersed  elements  will  become 
even  more  important.” 

A  key  theme  of  JV2010  is  the  emerging  importance  of 
“Information  Superiority.”  The  document  declares  that 
“we  must  have  information  superiority:  the  capability  to 
collect,  process,  and  disseminate  an  uninterrupted  flow 
of  information  while  exploiting  or  denying  an 
adversary’s  ability  to  do  the  same.” 

Global  Engagement 

The  next  step  in  the  test  technology  needs  identification 
process  was  to  look  at  USAF  visions  for  the  21" 
century.  The  post-Cold  War  “Global  Reach  -  Global 
Power"  vision  has  guided  the  USAF  for  the  past  6 
years.  Because  of  change  and  imcertainty,  a  new  vision 
was  required  to  carry  us  into  the  21"  century.  Therefore, 
in  response  to  the  uncertainties  of  the  post-Cold  War  era 
and  consistent  with  the  Chairman  of  the  Joint  Chiefs  of 
Staff  vision  of  “Full  Spectrum  Dominance,”  the  USAF 
developed  a  new  vision  to  guide  it  into  the  first  quarter 
of  the  21"  century.  Global  Engagement:  A  Vision  for  the 
2P^  Century  Air  Force  (Reference  2).  Global 
Engagement  will  exploit  the  strengths  of  modem  air  and 
space  power,  speed,  global  range,  stealth,  flexibility, 
precision,  lethality,  global/theater  situation  awareness, 
and  strategic  perspective  through  a  set  of  core 
competencies:  Air  and  Space  Superiority,  Global 
Attack,  Rapid  Global  Mobility,  Precision  Engagement, 
Information  Superiority,  and  Agile  Combat  Support. 
The  emerging  reality  of  the  21"  century  is  that,  largely 
through  the  strengths  of  Air  Power,  it  will  be  possible  to 
find,  fix  or  track,  target,  and  hit  anything  that  moves  on 
the  surface  of  the  earth. 

Global  Engagement  addresses  the  entire  USAF  -  people, 
capabilities,  and  inffastmcture  -  and  charts  the  course  of 
the  USAF  into  the  21"  century.  In  this  context.  Global 
Engagement  will  guide  the  evolution  of  the  USAF  fi-om 
an  “Air  and  Space  Force”  to  a  “Space  and  Air  Force” 
and  provide  a  focus  for  the  future  of  T&E.  However,  the 
vision  is  only  the  first  step  in  the  USAF’s  approach  to 
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long  range  planning.  Tlus  vision  establishes  overall 
direction,  but  the  USAF  needs  to  develop  a  long  range  plan 
to  make  fliis  vision  come  tme.  This  effort  has  begun,  but 
has  not  been  completed.  Thus,  any  use  of  Global 
Engagement  for  help  in  test  technology  needs  identification 
requires  a  liberal  interpretation  of  what  the  future  holds. 

Futuristic  Projections 

A  number  of  recently  published  futuristic  studies  and 
projections  address  the  next  30  years.  These  include 
D^ense  Science  and  Technology  Strategy  (Reference  3), 
New  World  Vistas:  Air  and  Space  Power  for  the 
Century  (Reference  4),  and  2025  Executive  Summary 
(Reference  5).  These  documents  address  futuristic 
projections  for  the  USAF  and  the  Office  of  the  Secretary  of 
Defense.  The  Defense  Science  and  Technology  Strategy 
(Reference  3)  along  with  three  support  documents:  the 
Basic  Research  Plan  (Reference  6),  Joint  Warfighting 
Science  and  Technology  Plan  (Reference  7),  and  Defense 
Technology  Area  Plan  (Reference  8)  initiated  a  corporate 
science  and  technology  planning  process  to  more 
effectively  link  products  of  the  Defense  Science  and 
Technology  program  with  the  needs  of  the  warfighter.  As 
such,  it  provides  information  on  technologies  the  science 
and  technology  community  is  working  on  that  tie  directly  to 
warfighter  requirements. 

New  World  Vistas  was  a  Secretary  of  the  Air  Force  and 
Air  Force  Chief  of  Staff  directed  study  accomplished  by 
the  Air  Force  Scientific  Advisory  Board.  They 
commissioned  this  study  to  identify  those  technologies 
that  would  guarantee  air  and  space  superiority  of  the 
U.S.  in  the  21"  century.  New  World  Vistas  documents 
enabling  technologies,  capabilities,  and  future  concepts 
and  systems  in  over  2,000  pages  in  15  volumes. 

2025  Executive  Summary  was  also  an  Air  Force  Chief  of 
Staff  directed  study,  but  conducted  by  the  Air  Force  Air 
University.  His  direction  was  to  “generate  ideas  and 
concepts  on  the  capabilities  the  U.S.  will  require  to 
possess  the  dominant  air  and  space  force  in  the  future.” 
The  study  included  use  of  a  forecasting  model  that 
addressed  key  drivers  to  define  a  strategic  planning 
space.  This  report  consisted  of  41  white  papers  on  more 
than  3,300  pages. 

These  three  studies  provide  a  wealth  of  information  of 
potential  future  systems  that  would  require  T&E. 
However,  there  has  been  no  effort  to  formally  and 
collectively  review  and  document  the  potential  impact 
to  USAF  T&E  of  the  projections  in  these  documents. 

Recent  Studies 

The  USAF,  during  the  period  1995  through  1998,  has 
participated  in  four  major  studies  (USAF  T&E  Mission 


Support  Plan  (MSP),  Tri-Service  Required  T&E 
Capabilities  Analysis  (RTECA),  USAF  Annual 
Planning  and  Programming  Guidance  (APPG),  and 
USAF  T&E  Requirements  Assessment  Team  (TERAT)) 
that  provide  some  insight  into  test  technology  needs  for 
the  21"  century.  We  will  discuss  these  studies 
independently  in  terms  of  their  objective,  the  scope  of 
the  effort,  process  used,  limitations,  and  a  brief 
summary.  An  attempt  will  then  be  made  to  take  the 
common  threads  between  the  studies,  and  propose  a  new 
conceptual  approach  to  identification  of  T&E  capability 
requirements,  and  thus,  test  technology  needs. 

Mission  Support  Plan  (MSP) 

Introduction.  The  USAF  vision  drives  the  Air  Force 
Modernization  Planning  Process.  The  Modernization 
Planning  Process  in  turn,  provides  the  corporate  plan  for 
leading  the  USAF  into  the  21"  centuiy.  The 
Modernization  Planning  Process  involves  a  strategy-to- 
task,  task-to-need,  and  need-to-solution  methodology. 
The  T&E  Mission  Support  Plan  (MSP)  was  developed 
as  a  part  of  the  Modernization  Planning  Process  by  a 
team  of  planners  fi-om  the  Single  Face  to  Customer 
offices  and  headquarters  personnel. 

Objective.  The  T&E  MSP  documents  how  T&E 
organizations  plan  to  meet  the  long  range  strategic  plans 
and  vision  of  the  USAF.  This  plan  differed  from  others 
in  that  it  took  a  long-range  view,  up  to  25  years,  and 
linked  strategies,  tasks,  needs  and  solutions  together  into 
a  plan  for  the  T&E  community. 

Scope.  The  MSP  is  the  primary  planning  document 
capturing  both  developmental  test  and  operational  test 
requirements  as  well  as  die  test  investments  needed  to 
modernize  the  test  infi-astmcture  in  order  to  provide  cost 
effective  and  efiBcient  support  to  current  and  future  weapon 
system  acquisitions.  The  T&E  MSP  was  developed  fi-om  an 
analysis  of  the  operational  command  Mission  Area  Plans, 
the  Science  and  Technology  -  Technology  Area  Plans,  Air 
Force  Operational  Test  and  Evaluation  Center’s  operational 
test  requirements,  test  needs  of  on-going  and  planned 
weapon  system  acquisitions,  and  the  strategic  vision  of  the 
U.S.  Department  of  Defense  (DoD).  Based  on  these 
documents,  T&E  developed  a  task  area  focus  to  provide  a 
corporate  view  toward  test  requirements  and  inJfrastructure 
modernization  planning.  Ihe  T&E  task  areas  were: 
Aircraft-Propulsion-Avionics,  Armament-Munitions,  C4I, 
Electronic  Warfare,  and  Space. 

Process.  Modernization  Planning  is  the  process  the  USAF 
uses  to  define  its  future  investment  roadmap.  Air  Force 
Policy  Directive  10-14,  Operational  Requirements  and 
Modernization  Planning  Process  (Reference  9)  and  Air 
Force  Instmction  10-601,  Air  Force  Mission  Needs  and 
Operational  Requirements  (Reference  10)  describes  this 
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process.  Accordingly,  the  T&E  MSP  development  team 
adhered  to  this  USAF  Modernization  Planning  Process. 

Since  Air  Force  Policy  Directive  10-14  specifically 
addresses  USAF  mission  areas  (such  as  Theater  Missile 
Defense,  Counter  Air,  Airlift,  etc.),  mission  support 
organizations  (such  as  T&E)  must  adapt  this  framework 
for  future  planning.  Thus,  mission  support 
modernization  planners  must  take  a  broader  perspective 
because  their  area  of  responsibility  is  to  support  all 
mission  areas  and  operations.  The  approach  involved 
an  Environmental  Assessment,  Mission  Support 
Assessment,  Mission  Support  Needs  Analysis,  and 
finally  development  of  a  MSP. 

The  first  step  involved  an  evaluation  of  the  environment 
in  which  the  mission  support  area  operates.  This  effort 
is  often  referred  to  as  strategic  planning  and  the 
resultant  strategic  assessment  provides  the  backdrop  to 
the  MSP.  For  our  effort  the  Environmental  Assessment 
included  reviews  of  the  visions  for  the  USAF  such 
as  Joint  Vision  2010  and  Global  Engagement. 
This  assessment  took  into  account  items  such  as 
customer  requirements,  budgetary  constraints/pressures, 
technology  opportunities,  and  organizational  structures. 
The  next  step  in  modernization  planning  involved  the 
strategy-to-task  process  as  its  principle  tool  to  identify 
the  tasks  required  to  accomplish  the  assigned  function. 
This  process  began  with  a  review  of  the  following 
documents.'  A  National  Security  Strategy  of  Engagement 
and  Enlargement  (Reference  11)  and  National  Military 
Strategy  of  the  United  States  of  America  1995 
(Reference  12),  which  yielded  operational  command 
deficiencies,  which  in  turn,  provided  key  inputs  into  the 
identification  of  T&E  support  functions.  This 
information,  along  with  the  science  and  technology 
community  requirements  from  their  Technology  Area 
Plans,  and  the  developmental  and  operational  test 
requirements  of  planned  and  on-going  weapon  system 
programs  formed  the  basis  for  the  T&E  Mission  Support 
Assessment.  After  the  Mission  Support  Assessment  task 
identification,  the  team  conducted  a  Mission  Support 
Needs  Analysis  using  the  task-to-needs  process.  This 
process  assessed  whether  current  T&E  capabilities  could 
adequately  satisfy  the  tasks  as  identified  in  the  Mission 
Support  Assessment.  If  not,  the  team  documented  the 
deficiencies. 

The  final  element  of  modernization  planning,  the  MSP, 
summarized  and  used  the  products  of  the  Mission 
Support  Assessment  (tasks)  and  Mission  Support  Needs 
Analysis  (deficiencies)  to  identify  key  technologies  and 
resource  modernization  efforts  required  to  correct 
identified  deficiencies.  The  MSP  product  is  a 
modernization  roadmap.  The  T&E  MSP  serves  as  the 
primary  planning  document  and  provides  the  proper 


focus  for  limited  investment  dollars.  Leading  up  to  this 
step,  the  T&E  MSP  team  operated  in  a  fiscally 
unconstrained  environment.  During  this  step,  the  team 
evaluated  and  prioritized  the  proposed  investments  and 
developed  funding  strategies.  In  the  near-term  planning 
horizon  (2  to  6  years),  the  T&E  MSP  process  resulted  m 
a  detailed  investment  project  list  for  each  task  area  with 
planned  funding.  In  the  mid-term  planning  horizon  (7  to 
16  years)  the  MSP  process  identifies  key  modernization 
areas  the  USAF  T&E  infrastructure  must  concentrate  on 
to  address  the  identified  long-term  deficiencies.  In  the 
far-term  (out  to  25  years),  the  T&E  MSP  provides  the 
vision  of  the  future.  As  such,  the  MSP  linked  the 
requirements  identification  process  with  the  investment 
and  budgeting  process. 

Limitations.  A  primary  limitation  of  the  T&E  MSP 
came  from  the  fact  that  the  USAF  vision  was  fiscally 
unconstrained.  This  meant  that  the  tasks,  needs  and 
solutions  defined  in  the  T&E  MSP  were  also 
unconstrained.  Decisions  on  what  solutions  should  and 
should  not  be  funded,  that  support  program  plans 
identified  in  the  out-years,  were  difficult  because  no  one 
knows  whether  the  program  will  be  funded  or  not. 

From  a  procedural  perspective,  additional  limitations 
were  numerous.  This  was  the  first  MSP  developed,  as 
such;  the  process  was  not  entirely  efficient.  The 
timeframe  allotted  to  accomplish  the  task  was  short,  and 
the  manpower  was  limited.  In  addition,  application  of 
the  modernization  process  as  it  relates  to  development 
of  the  MSP  was  a  learning  process.  We  understood  the 
basic  task,  but  had  to  determine  what  the  input  sources 
were,  where  to  obtain  the  warfighter  deficiencies,  the 
military  strategies,  etc,  and  had  very  little  direction  from 
higher  headquarters  on  what  the  MSP  was  to  be. 
Therefore,  the  team  had  to  develop  the  MSP  from  the 
ground  up. 

Summary.  The  first  edition  of  the  T&E  MSP  was  a 
good  attempt  at  converting  strategies-to-task-to-need-to- 
solution.  It  would  have  been  a  better  product  if  the 
reviews  of  the  strategies  had  been  an  on-going  process 
all  year  arid  not  a  quick  look  when  the  T&E  MSP  was 
being  developed.  However,  it  still  provided  excellent 
input  into  the  Air  Force  Annual  Planning  and 
Programming  Guidance.  The  MSP  effort  as  a  whole 
provided  one  of  the  best  looks  at  T&E  capability 
requirements  accomplished  to  date. 

Required  Test  and  Evaluation  Capabilities 
Assessment  (RTECA) 

Introduction.  The  1996  Defense  Authorization  Bill, 
Section  277,  directed  the  Secretary  of  Defense  to 
develop  a  5-year  plan  to  consolidate  and  restructure  the 
laboratories  and  T&E  centers  of  the  U.S.  DoD.  The 
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Office  of  the  Secretary  of  Defense  termed  this  objective 
as  Vision  21,  The  Plan  for  Century  Laboratories 
and  Test  and  Evaluation  Centers  of  the  Department  of 
Defense  (Reference  13).  Vision  21  highlighted  how  the 
U.S.  DoD  planned  to  restructure,  reduce  and  revitalize 
the  laboratory  and  T&E  infrastructure.  The  T&E  portion 
of  Vision  21  was  led  by  the  Service  Vice-Chiefs  in  their 
roles  as  the  Board  of  Directors  for  the  T&E  Executive 
Agent  Structure,  and  subsequently  tasked  to  the  T&E 
Reliance  Investment  Board  (TERIB)  to  conduct 
this  study. 

As  one  portion  of  the  Vision  21  effort,  the  Required  Test 
and  Evaluation  Capabilities  Analysis  (RTECA) 
supported  Vision  21  by  identifying  the  future  required 
T&E  capabilities  from  a  T&E  perspective.  Multiple 
organizations  participated  in  this  process  (field, 
command,  headquarters,  and  contractor  graybeards). 
However,  the  Tri-Service  Reliance  Panels  conducted  the 
bulk  of  the  effort  with  limited  contractor  support. 

Objective.  The  objective  of  RTECA  was  to  document 
required  T&E  capabilities  for  the  near-term  (1996  to 
2005)  and  the  far-term  (2006  to  2020)  and  to  show  the 
linkage  of  required  test  capabilities  to  the  documented 
warfighter  needs. 

Scope.  This  was  a  massive,  labor  intensive  effort 
involving  the  three  U.S.  services  (Air  Force,  Army  and 
Navy)  and  support  from  the  Joint  Program  Office  for 
Test  and  Evaluation  (JPO[T&E]).  The  ramp-up  of 
qualified  personnel,  composed  of  service  professionals 
and  experienced  consultants,  in  a  relatively  short 
timeframe  was  a  significant  accomplishment.  A  new, 
innovative  approach  was  necessary  that  required 
identification  and  gathering  of  source  documents  and 
the  development  of  tools  for  research  and  data 
collection.  The  T&E  capability  requirements  were 
extracted  explicitly  or  derived  from  the  source 
documents  used  and  then  entered  into  the  database.  This 
effort  encompassed  the  fusion  of  intuitive  knowledge 
from  experts  across  all  Reliance  Areas  of  Land  Combat, 
Sea  Combat,  Air  Combat,  Space  Combat,  Armament 
and  Munitions,  Electronic  Combat,  C4I,  Targets,  Test 
Environments,  and  Common  Range  Instrumentation. 
The  Service’s  Operational  Test  Agencies  also  developed 
a  functional  area  summary  for  the  RTECA  effort  as  it 
applied  to  Operational  Test  and  Evaluation.  The  first 
priority  of  the  RTECA  effort  was  to  address  the  focus 
areas  of  the  Vision  21  T&E  Study  Plan:  Air  Combat, 
Space  Combat,  Armament/Munitions,  C4I,  and 
Electronic  Combat.  The  remaining  focus  areas  (Sea 
Combat,  Land  Combat,  Common  Range 
Instrumentation,  Targets,  and  Test  Environments) 
comprised  the  secondary  priorities  of  the  RTECA  effort. 


Process.  The  Reliance  panels  and  expert  consultants 
defined  the  structure  for  identifying  T&E  requirements. 
This  structure  provided  a  method  to  document  the 
required  T&E  capabilities  in  a  manner  identifiable 
within  the  T&E  community.  Although  this  structure  was 
to  apply  to  all  the  Reliance  Areas  examined  by  the 
RTECA  study,  there  were  some  inconsistencies  between 
panels  on  how  they  used  this  structure.  This  structure 
was  simply: 

•  Reliance  Area 

•  Physical  Characteristics 

•  Technical  Characteristics 
•  Types  of  Test 

•  T&E  Capability  Characteristics 
•  Attributes 

•  Human  Characteristics 

The  physical  characteristics  was  exactly  that,  size, 
shape,  etc.  For  the  technical  characteristics,  the  team 
developed  and  defined  types  of  tests,  T&E  capability 
categories  and  attributes  to  describe  test  capabilities 
required  by  the  T&E  community  in  order  to  assure  the 
needs  of  the  warfighter  were  met.  They  then  grouped, 
organized,  and  inserted  these  types  of  tests,  T&E 
capability  categories,  and  attributes  into  a  database 
under  which  T&E  capability  requirements  would 
subsequently  be  entered.  The  human  characteristics 
were  the  requirements  for  personnel  in  reference  to 
experience,  skills,  etc. 

The  IPO(T&E)  researched,  collected,  and  provided  the 
focus  area  teams  with  a  compendium  of  over  300 
available  source  documents,  which  identified  concepts, 
systems,  and  technologies  to  meet  the  needs  of  the 
warfighter.  The  focus  area  teams  supplemented  these 
documents  with  other  program-specific  planning 
documents  such  as  Mission  Need  Statements, 
Operational  Requirements  Documents,  Program 
Management  Directives,  and  Test  and  Evaluation 
Master  Plans.  In  deriving  the  required  T&E  capabilities, 
the  RTECA  team  members  fused  the  information  from 
the  available  documents  with  their  own  expert 
knowledge  of  warfighter  requirements  and  T&E 
technologies.  Reliance  Area  panels  identified  major  war 
fighting  systems,  which  would  likely  employ  these 
technology  drivers  and  subsequently  distilled,  primarily 
by  intuition,  major  T&E  capability  requirements 
believed  to  be  of  significance.  These  specific  T&E 
capabilities  necessary  to  test  and  evaluate  systems 
fielded  or  likely  to  be  fielded  were  entered  into  the 
database,  linked  directly  to  source  documents.  This 
database  was  the  primary  product  of  the  RTECA  effort. 

Limitations.  Despite  the  significant  effort  and  resource 
expenditure  by  the  JPO(T&E)  and  the  Reliance  teams 
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prior  to  the  document  review  and  data  collection  effort, 
there  were  significant  limitations  and  impediments  with 
the  RTECA  process.  Examples  of  some  limitations  are 
provided  here  to  help  put  the  results  of  this  effort  in 
perspective. 

•  Lack  of  critical  program  documents/lack  of 
classified  documents 

•  Insufficient  time  to  accomplish  study 

•  No  prioritization  of  identified  requirements 

•  Varied  level  of  detail  on  requirements  based  on 
document  source 

•  Information  organization/taxonomy  sub- 
optimal 

•  Minimal  analytical  ability  from  database 

•  Lack  of  guidance/intended  use  of  the  resultant 
database 

•  Impact  potential  of  new  technology  on  required 
new  test  capabilities  not  addressed 

Summary.  In  general,  the  purpose  behind  RTECA  was 
good,  but  the  actual  process  of  developing  the  database 
was  flawed.  The  product  of  RTECA,  the  nearly  14,000 
records  in  the  RTECA  database,  documented  a  large 
percentage  of  the  T&E  capability  requirements. 
However,  the  teams  did  not  consider  RTECA  to  be  a 
100-percent  solution,  but  an  important  first  step  to  tie 
required  T&E  capabilities  to  warfare  requirements.  It  is 
clear  that  RTECA  was  a  process  with  significant  future 
utility.  The  RTECA  requires  some  evolution  in  order  to 
become  a  more  useful  tool  to  provide  a  user  link  from 
warfighter  needs  and  technology  drivers  to  required 
T&E  capabilities. 

The  U.S.  DoD  should  use  caution  when  attempting  to 
make  reduction,  restructuring,  and  revitalization 
decisions  based  on  this  database.  The  limitations  of  the 
database,  previously  mentioned,  should  be  considered, 
as  well  as  the  probability  that  there  will  be  programs, 
either  classified  or  not  yet  identified,  which  will  drive 
significant  technological  changes  to  our  required  tri¬ 
service  T&E  capabilities.  The  database,  derived 
primarily  from  current  programmatic  documents,  is 
perishable  in  nature.  While  it  is  probably  adequate  for 
aiding  near-term  Vision  21  decision  making,  unless 
updated,  this  database  should  not  be  used  to  define  T&E 
range  and  facility  investments  in  the  out-years. 

Annual  Planning  and  Programming  Guidance 
(APPG) 

Introduction.  The  USAF  long  range  plan  provides  for 
two  end  states  for  the  T&E  community  to  achieve.  They 
are: 

End  State  1:  A  T&E  infrastructure  that  maintains 

core  USAF  T&E  expertise  in  all  five  T&E 


mission  areas  (aircrctft/propulsion/avionics, 
electronic  combat,  space,  armament/munitions, 
and  command/control/communication/computers 
and  intelligence)  at  a  reduced  cost  The 
infrastructure  need  not  be  USAF  owned  and  the 
USAF  should  eliminate  any  unnecessary  overlap 
and  redundancy  with  other  DoD,  Federal  and/or 
commercial  test  facilities.  This  will  be  achieved 
by  the  year  2005. 

End  State  2:  Use  of  test  validated  modeling 
and  simulation,  along  with  field  testing, 
will  be  a  key  element  in  evaluating 
performance/effectiveness,  suitability,  and  life 
cycle  sustainment.  This  will  be  achieved  by  the 
year  2015. 

Objective.  A  team  of  planners  from  the  Single  Face  to 
Customer  test  center  and  headquarters  offices  was 
formed  to  create  a  T&E  modernization  working  group. 
This  T&E  modernization  working  group  was  tasked  to 
develop  the  portion  of  the  USAF  Aimual  Planning  and 
Programming  Guidance  (APPG)  dealing  with  the 
modernization  requirements  to  accomplish  the  two  T&E 
end  states. 

Scope.  The  T&E  modernization  working  group  looked 
out  to  the  year  2025  to  determine  types  of  test 
capabilities  required  to  support  weapons  testing.  This 
required  reviewing  the  long  range  plaiming  and  vision 
documents  that  the  USAF  had  produced  in  recent  years. 
Using  the  T&E  expertise  represented  on  the  working 
group,  a  timeline  for  when  a  T&E  capability  would  need 
to  come  online,  and  a  rough  order  of  magnitude  of 
cost  for  that  capability  were  developed.  The  T&E 
modernization  woridng  group  incorporated  this  inteipretation 
into  the  APPG  and  die  USAF  long-range  plan. 

Process.  The  T&E  modernization  working  group  used  a 
multi-step  process  in  developing  the  T&E  USAF  Long 
Range  Modernization  Plan  for  the  two  end  states.  As 
step  one,  the  working  group  performed  a  documentary 
scan  of  the  environment  in  which  USAF  T&E  will 
operate  within  the  near-  (2000  to  2005),  mid-  (2005  to 
2015)  and  far-  (2015  to  2025)  term  timeframes.  The 
first  items  reviewed  were  National  and  USAF  military 
strategy  with  a  vision  of  providing  global  power  and 
reach  through  air  and  space  superiority,  global  attack, 
rapid  global  mobility,  precision  engagement, 
information  superiority,  and  agile  combat  support.  The 
scan  thus  addressed  initiatives  within  the  acquisition 
reform  process,  which  T&E  supports,  such  as 
acquisition  cycle  time  reduction,  life  cycle  cost 
reduction  and  weapon  system  technology  protection. 
The  group  also  reviewed  visionary  documents  defining 
the  USAF  of  the  future,  such  as  New  World  Vistas  and 
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AF  2025.  Finally,  the  scan  addressed  initiatives/policies 
mandated  on  the  T&E  community  by  its  stakeholders, 
such  as:  guidance  that  the  USAF  must  lead/own  its  core 
T&E  capabilities,  while  at  the  same  time  reducing 
infrastructure  and  T&E  budgets;  guidance  that  T&E 
must  rely  more  on  modeling  and  simulation,  hybrid  test 
capability  resources,  and  electronic  linking  of  test 
locations;  and  guidance  for  outsourcing/privatization  of 
testing.  From  this  scan,  a  vision  for  USAF  T&E 
modernization  that  encompasses  the  two  T&E  end  states 
was  created: 

Vision 

Provide  an  integrated  T&E  complex  without 
geographical  boundaries  that  provides  a 
national  capability  for  T&E  of  air  and  space 
weapon  systems  utilizing  world-class  facilities 
within  government  and  commercial  sectors. 

The  vision  recognized  that  the  USAF  must  rely  on 
other  government  and  commercial  sector  test 
capabilities  when  appropriate  so  that  imnecessary 
modemization/duplication  of  USAF  T&E  infrastructure 
will  not  occur. 

Step  two,  from  this  vision,  was  development  of  five 
modernization  strategies  to  provide  an  overarching 
framework  to  guide  T&E  investments  in  the  future. 
These  T&E  Strategies  were:  grow  constructive  and 
virtual  capabilities  (modeling  and  simulation);  sustain 
core  T&E  capabilities;  reduce  cost/improve  efficiencies; 
exploit  existing  assets  to  meet  new  T&E  requirements; 
and  pursue  technology  for  T&E  innovations. 

The  T&E  strategies  recognized  the  increased 
significance  of  modeling  and  simulation  and  linking 
geographically  separated  facilities.  The  strategies  also 
provide  for  maintaining/sustaining  that  “core” 
infrastructure  envisioned  being  required  through  the 
year  2025  and  vital  to  development  and  sustainment  of 
U.S.  DoD  systems.  In  addition,  the  strategies  recognize 
that  with  decreasing  budgets  for  T&E  modernization 
accounts,  plus  decreasing  test  budgets  in  weapon  system 
development  programs,  testing  must  become  more 
affordable  through  cost  reductions  and  improved 
business/engineering  practices. 

In  step  three,  the  group  developed  a  time-phased 
roadmap  of  future  T&E  investments  in  alignment  with 
these  five  strategies.  This  roadmap  was  baselined  on  the 
current  near-term  investments/program  developments, 
projections  of  future  (mid-  and  far-term)  weapon  system 
development  programs  and  their  associated  technology 
drivers. 


The  last  step  for  the  modernization  working  group  effort 
was  development  of  a  rough  order  of  magnitude 
estimate  for  funds  required  to  support  the  USAF  T&E 
roadmap.  The  near-term  funding  profile  reflected  the 
current  modernization  efforts  underway.  Recommended 
mid-term  investments  captured  funding  required  to 
sustain  core  T&E  capabilities,  an  increase  to  modeling 
and  simulation  capabilities  to  meet  T&E  end  state  2,  and 
funds  to  prepare  the  USAF  to  test  the  weapon  concepts 
that  were  forecasted  for  the  mid-  to  far-term  timeframe. 
Far-term  investment  profiles  for  the  five  strategies 
reflected  that  major  investments  would  have  to  be  made 
in  the  mid-term  timeframe  and  that  the  investments  in 
the  fiscal  year  2015  to  2025  timeframe  were  for 
sustainment  of  those  capabilities  and  identification  of 
those  new  technologies  in  beyond  the  far-term. 

Limitations.  The  primary  limitations  to  this  effort  were 
the  short  time  period  to  accomplish  it  and  the  lack  of  an 
established  process  for  inputs.  An  in-depth  look  into 
requirements  was  not  possible.  Earlier  one-time  efforts 
such  as  RTECA  and  Ae  MSP  were  used  as  inputs  on 
T&E  requirements.  The  lack  of  an  established  process  to 
provide  T&E  interpretation  of  higher  level  guidance  or 
to  provide  warfighter  and  science  and  technology 
requirements  and  thus,  T&E  requirements,  heavily 
impacted  the  validity  of  the  modernization  working 
group’s  efforts. 

Although  the  modernization  working  group  developed  a 
macro-level  long-range  T&E  roadmap  as  a  product  for 
this  study  effort,  the  group  realized  that  this  was  just  a 
start  for  what  is  required  to  meet  the  two  defined  T&E 
end  states  in  the  desired  timeframe.  There  was 
agreement  in  that  the  pursuit  of  these  end  states  would 
provide  the  USAF  with  an  evolving  capability  to 
evaluate  new  and  existing  system’s  performance  in  a 
more  cost  effective  and  timely  manner.  However,  the 
USAF  T&E  community  cannot  achieve  these  end  states 
in  isolation.  There  must  be  a  commitment  on  part  of  the 
USAF,  in  partnership  with  industry,  to  invest  in  T&E 
capability.  To  avoid  duplicating  existing  T&E 
capabilities  at  contractor  locations,  the  USAF  must  be 
aware  of  planned  and  existing  contractor  T&E 
capabilities  (insight  into  industry  planning  ciurently 
does  not  exist).  The  working  group  recommended 
establishment  of  a  formal  and  enduring  process  to 
partner  with  the  private  sector  in  long-range  investment 
planning. 

Summary.  The  strategic  area  emphasized  least  in  the 
near-term  is  in  development  of  new  test  technologies. 
Based  on  the  limited  documentary  scan,  the  USAF 
should  invest  in  test  technology  development  focusing 
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on  areas  where  test  capability  is  seriously  insufficient. 
These  investments  are  necessary  to  allow  study  of  the 
requirement  for  new  test  technology,  versus  actually 
developing  the  capability.  Actual  development  would 
occur  in  the  mid-  or  far-term.  While  investments  in 
these  areas  can  be  debated,  they  should  be  based  on 
common  threads  fi-om  all  of  the  long-range  planning 
documents  reviewed  as  part  of  a  formal  process. 

The  majority  of  the  investment  dollars  in  the  mid-term 
will  address  T&E  required  for  the  next  generation 
weapon  systems.  Mid-term  investments  are  based  on 
predictions  that  the  USAF  is  evolving  fi-om  an  “Air  and 
Space  Force”  into  a  “Space  and  Air  Force.”  These 
investments  are  dependent  on  having  previously 
invested  in  test  technology  development  in  the  near- 
term  timefi-ame  and  will  impact  future  investments  for 
the  far-term. 

Investments  for  the  far-term  are  the  modernization 
working  group  interpretations  of  T&E  requirements 
based  on  historical  T&E  investments  and  limited  review 
of  visionary  docmnentation.  Predictions  are  that 
modeling  and  simulation  will  remain  a  large  portion  of 
T&E  and  by  the  far-term  may  consume  a  larger  portion 
of  the  investment  budget.  The  far-term  will  require 
consistent  refocusing,  as  new  forecasts  of  future 
requirements  become  available. 

In  general,  in  the  near-term,  the  requirements  are  very 
well  defined,  while  the  focus  becomes  less  clear  in  the 
mid-term,  and  turns  into  a  fog  for  the  far-term.  The 
focus  for  the  future  should  be  on  the  mid-term 
requirements,  as  this  is  the  area  we  can  impact  the  most. 
As  the  mid-term  timeframe  approaches,  the  fog  should 
begin  to  clear  on  those  far-term  investments  now 
defined,  while  new  far-term  investments  are  identified. 

Test  and  Evaluation  Requirements  Assessment 
Team  (TERAT) 

Introduction.  With  the  results  of  the  above  study 
efforts  and  evaluations  in  hand,  senior  USAF  leadership 
realized  that  there  was  still  a  missing  part  to  the  puzzle 
of  “right-sizing”  the  T&E  infi-astructure,  the  program 
directors  perspective.  The  T&E  community  completed 
all  of  the  previous  efforts  looking  at  T&E  requirements 
and  the  results  are  from  their  perspective.  It  became 
obvious  that  to  make  the  best  decisions  on  the  future  of 
the  USAF  T&E  infrastructure,  the  systems  acquisition 
community  needed  to  provide  their  perspective  to  the 
future  T&E  requirements  picture. 

The  Test  and  Evaluation  Requirements  Assessment 
Team  (TERAT)  is  an  ongoing  effort  designed  to  provide 
both  the  systems  acquisition  and  T&E  communities’ 
perspectives  on  the  future  requirements  for  T&E 


capabilities.  As  in  previous  studies,  multiple 
organizations  are  participating  in  the  process.  The 
biggest  difference  is  that  this  effort  is  co-sponsored  by 
the  USAF  Secretary  of  the  Air  Force  Acquisition  Chief 
(SAF/AQ)  and  the  USAF  T&E  Chief  (AF/TE).  In 
addition,  a  general  officer  fi-om  the  USAF  acquisition 
community  is  in  charge  of  the  team,  thus  truly  putting 
the  focus  on  the  acquisition  side  of  the  equation.  The 
key  products  of  this  study  effort  are  a  comprehensive 
list  of  USAF  Programs  (with  points  of  contact),  a 
maturing  database  of  test  requirements,  data  query  tools, 
and  some  data  analysis  for  the  “decision  makers.” 

Objective.  The  objective  of  TERAT  is  to  assess  and 
document  T&E  capabilities  and  capacity  required  for 
USAF  modernization  now  through  the  year  2010. 

Scope.  As  with  the  previous  studies,  TERAT  is  a  massive 
labor-intensive  effort.  The  use  of  the  knowledgeable 
mission  area  experts  from  the  RTECA  study  has  been 
crucial  to  the  success  of  this  effort.  The  TERAT  is  limited 
to  the  USAF  at  this  time,  but  may  eventually  expand  to 
include  all  three  U.S.  services  (Army,  Air  Force  and  Na-vy). 
TERAT  has  the  luxury  of  learning  from  and  applying 
concepts  used  in  previous  studies.  Specifically,  TERAT 
utilized  to  a  great  extent  the  methodology  developed  for 
RTECA  in  the  development  of  the  tools  and  structure  for 
research  and  data  collection.  The  T&E  capabilities 
requirements  (TECR)  methodology,  based  on  portions  of 
the  RTECA  effort,  was  the  foundation  for  TERAT.  As  with 
the  MSP  effort,  the  following  mission  areas  were  the  focus 
of  the  effort;  Aircraft-Porpulsion-A-vionics,  Armament- 
Munitions,  C4I,  Electronic  Warfare,  and  Space. 

With  the  focus  of  the  TERAT  effort  on  the  acquisition 
side  of  the  equation,  a  large  part  of  the  effort  was 
developing  a  list  of  all  USAF  programs,  to  query  and 
acquire  data  from,  on  T&E  requirements.  This  effort 
alone  was,  and  still  is,  a  huge  task,  not  because  of  the 
large  niunber  of  programs,  but  to  ensure  capture  of 
100  percent  of  USAF  programs.  As  of  this  date,  nearly 
450  weapon  system  acquisition  programs  are  a  part  of 
the  overall  TERAT  database. 

Through  the  use  of  the  T&E  capabilities  requirements- 
based  methodology,  a  data  query  was  sent  to  these 
programs  and  data  received  back  from  nearly  all  of 
them.  One  of  the  key  items  discovered  in  completing 
this  data  call  was  that  the  program  managers  needed 
more  information  on  specific  T&E  capabilities  that  they 
could  reference  to  meet  their  needs.  In  a  TERAT 
follow-on  effort,  a  data  call  to  T&E  facility/capability 
managers  requested  specific  information  on  test 
capabilities  that  will  be  made  available  to  the  program 
managers.  As  part  of  this  TERAT  follow-on  effort,  a 
second  data  query  will  be  sent  to  the  program  managers 
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with  the  goal  of  getting  them  to  provide  very  focused 
future  T&E  requirements  down  to  the  facility  level. 

Process.  TERAT  is  a  two-phase  effort.  Phase  I  included 
the  adaptation  of  the  T&E  capabilities  requirements- 
based  methodology  to  the  specific  goals  of  TERAT, 
development  of  a  target  list  of  programs  to  query,  and 
the  actual  data  call  to  these  programs  asking  for  their 
T&E  requirements.  Phase  II  consists  of  refinement  of 
the  data  query  tools,  a  data  call  to  the  facility/capability 
managers,  and  a  second  data  call  to  the  program 
managers  for  T&E  requirements  to  the  facility  level. 

Throughout  the  entire  TERAT  process,  the  mission  area 
experts,  representatives  from  SAF/AQ,  AF/TE,  Air 
Force  Materiel  Command,  and  the  ANSER  Corporation 
have  been  developing  and  refining  the  study  structure. 
Currently  the  structure  is: 

•  Program 

•  Mission  Area 

•  T&E  Capability  Requirements 

•  T&E  Resources  Categoiy  (Digital  Modeling, 
Ground  Test,  Open  Air  Range) 

•  Source  (Air  Force,  Army,  Navy,  Commercial, 
Foreign) 

•  Facility/Asset/Capability  Class 

•  Facility/Asset/Capability 

•  Demand  (Months) 

This  structure  represents  the  hierarchy  and  outline  for 
the  database  that  forms  the  basis  for  die  entire  TERAT 
effort.  The  designed  structure  allows  the  individual 
program  manager  to  work  his/her  way  down  by 
selecting  the  appropriate  category  under  each  major 
area.  The  specific  test  facility/capability  information  is 
available  in  the  facility/asset/capability  category  and  at 
this  point  the  program  manager  can  identify  future  T&E 
requirements  for  his/her  program.  Multiple  paths  and 
entries  are  acceptable  and  encouraged  to  ensure 
maximum  capture  of  all  T&E  requirements. 

Limitations.  Again,  as  with  the  other  studies,  despite 
the  significant  effort  and  resource  expenditures,  there 
are  numerous  limitations  and  impediments  to  the 
TERAT  process.  The  major  limitations  are: 

•  Very  high  level  look  -  some  programs  and 
requirements  overlooked 

•  Current  program  focus  limits  long-term  T&E 
requirements  identification 

•  Concurrent  survey  process  and  analysis 
development 

•  Possible  duplication  of  inputs 

•  Short  timeframe  to  complete  effort 
compromises  data  quality  and  completeness 


•  Limited  capacity  for  response  review 

•  Data  not  thoroughly  reviewed  before  entry  into 
database 

•  Software  application  and  survey  tool 
shortcomings 

•  Program  target  list  incomplete  and  difficult  to 
manage 

Summary.  The  TERAT  certainly  has  accomplished  its 
goal  of  identifying  T&E  requirements  for  the  near-  and 
mid-term  out  to  the  year  2010.  With  the  focus  on 
identifying  these  requirements  from  the  consumer 
(acquisition/program)  perspective,  TERAT  has 
tremendous  credibility.  However,  as  with  the  other 
studies,  TERAT  is  not  the  complete  solution.  Taken 
with  the  other  studies,  these  efforts  are  closing  in  on 
providing  the  complete  picture  to  the  “decision  makers” 
on  what  the  USAF  and  U.S.  DoD  T&E  infrastructure 
should  look  like.  TERAT  is  a  “tool”  that  is  still  evolving 
and  will  most  likely  be  instituted  as  a  yearly  or  once 
every  18  month  process  to  help  in  the  T&E 
infrastructure  investment  process. 

The  danger  in  TERAT  is  the  use  of  this  data  to  make 
major  T&E  reduction  or  investment  decisions.  With  the 
limitations  previously  identified,  the  probability  of 
making  the  best  decision  based  on  this  data  is  not  very 
high.  In  addition,  the  “right-sizing”  decisions  for  the 
T&E  infrastructure  must  look  out  beyond  the  year  2010 
for  which  TERAT  is  limited  to. 

The  TERAT  has  provided  a  tremendous  snapshot  and 
analysis  of  T&E  requirements.  This  effort  must  be 
integrated  with  the  futuristic  projections  as  mentioned 
earlier  m  this  paper  as  well  as  the  future  warfighter 
requirements.  Only  then  will  a  clear  picture  be 
developed  that  will  allow  for  the  major  T&E  reduction 
and  investment  decisions  to  be  made  with  a  high  level 
of  confidence. 

Proposed  New  Process 

The  four  studies  discussed  above  show  the  USAF  has  been 
working  diligently  for  the  past  several  years  to  identify 
T&E  capability  requirements  in  preparation  for  the  21“ 
century.  However,  as  can  be  seen  from  these  discussions, 
tiiere  is  no  single  process  for  T&E  capability  requirement  or 
test  technology  needs  identification  in  the  USAF.  There 
needs  to  be  a  formalized  test  technology  needs 
identification  process  to  provide  for  integrating  existing  and 
future  warfighter  requirements  and  strategic  guidance  into 
identifiable  T&E  capability  requirements,  which  in  turn, 
would  allow  for  identification  and  discussion  of  new  test 
technology  needs  for  21“  century  USAF  T&E. 

We  do  not  necessarily  propose  that  any  of  the  discussed 
studies  should  or  should  not  be  retained  in  their  entirety 
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as  parts  of  the  new  process,  nor  do  we  attempt  to  state 
whom  should  carry  out  any  or  all  of  the  elements  of  this 
notional  process.  However,  our  process  may  incorporate 
portions  of  previous  methods  used  during  some  of  the 
discussed  studies  if  they  appear  to  fit  within  the  process 
and  were  considered  successful.  Our  objective  is  merely 
to  outline  a  conceptual  process  (Figure  14-1)  for 
providing  a  systematic  approach  to  T&E  capability 
requirement  and  test  technology  needs  identification. 
This  process  is  designed  around  the  basic  “strategy-to- 
task-to-need-to-solution”  approach  as  outlined  in  our 
USAF  Modernization  Planning  Process.  In  the 
“strategy-to-task”  portion  of  the  process,  we  have 
basically  two  elements:  technology  push/warfighter  pull 
and  T&E  synthesis/linkage.  The  technology 
push/warfighter  pull  element  is  fairly  well  established. 
The  Major  Commands  identify  their  warfighter 
requirements  and  deficiencies  in  their  Mission  Area 


Plans  based  on  the  strategic  visions  and  guidance  as 
previously  discussed.  The  Science  and  Technology 
community  then  prepares  their  Technology  Area  Plans 
identifying  their  proposed  technology  roadmaps  based 
on  these  warfighter  requirements  and  deficiencies  and 
potential  technological  applications.  This  process  is 
currently  accomplished  by  Technology  Plaiming 
Integrated  Product  Teams  (TPIPTs)  chaired  by  Product 
Center^  long-range  planning  personnel  and  including 
personnel  from  the  research  laboratories,  operational 
community,  air  staff,  logistics,  finance,  intelligence,  and 
test  community.  However,  the  output  of  the  TPIPTs 
does  not  include  identification  of  technology  roadmaps 
for  test  capability  requirements,  only  for  warfighter 
requirements.  Therefore,  the  USAF  needs  to  formally 
implement  the  second  element  in  the  “strategy-to-task” 
portion  of  the  proposed  process,  T&E  synthesis/linkage. 


STRATEGY-TO-TASK 


Figure  14-1.  Notional  T&E  Requirement  and  Test  Technology  Needs  Identification  Process 


^  Aeronautical  Systems  Center  (ASC),  Electronic  System  Center 
(ESC),  and  Human  Systems  Center  (HSC) 
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The  T&E  synthesis/linkage  is  basically  an  interpretation  of 
the  information  obtained  from  the  push/pull  element  of  the 
strategy-to-task  portion  of  the  process.  A  common  theme 
from  the  recent  studies  was  the  need  to  interpret  or 
synthesize  die  higher  level  visionaiy  documents  and  long- 
range  plans.  Few  of  the  visionary  documents  or  long-range 
plans  previously  discussed  address  T&E  specifically  {New 
World  Vistas  being  the  exception).  The  T&E  community 
needs  to  formalize  this  portion  of  the  process  to  develop  a 
recurring  review  of  these  types  of  documents  to  provide 
planners  the  information  required  to  accomplish  strategic 
planning  for  T&E.  Several  of  the  studies  discussed  in  this 
paper  attempted  some  form  of  interpretation  of  these  types 
of  documents,  but  with  limited  scope,  time,  and  resources. 
Establishment  of  a  formal  group  to  accomplish  this  task 
routinely  as  part  of  the  T&E  modernization  planning 
process  is  mandatory.  Accomplishing  this  ^e  of 
review/interpretation  on  an  ad-hoc  basis  does  not  provide 
the  consistency  and  continuity  required  of  a  strategic 
planning  process.  In  order  for  this  “strategy-to-task”  phase 
of  the  proposed  process  to  work,  sufficient  time  must  be 
provided  to  collect,  review,  analyze  and  report  on  the 
interpretation/synthesis  of  this  tremendous  amount 
of  information. 

The  next  phase  of  the  process,  task-to-need,  develops 
the  actual  T&E  requirements  (i.e.,  what  T&E  tasks  we 
need  to  accomplish).  Two  of  the  studies  we  participated 
in,  RTECA  and  TERAT,  were  good  attempts  at  deling 
T&E  requirements  from  the  T&E  community  and 
acquisition  community.  Neither  community  has  all  the 
answers  when  it  pertains  to  required  testing.  The  testers 
are  viewed  as  wanting  to  overkill  the  testing  required, 
and  the  acquisition  community  is  viewed  as  wanting  to 
accomplish  little  or  no  testing  to  save  costs.  Collectively 
though,  they  can  identify  the  anticipated  T&E 
requirements.  By  matching  their  inputs,  the  strategic 
planners  can  get  the  entire  picture  of  where  we  need  to 
go  in  the  21®'  century.  As  with  the  previous  phase  of  the 
process,  a  formal  group  needs  to  be  established  to  direct 
these  two  efforts.  While  neither  RTECA  nor  TERAT 
were  considered  the  best  example  of  how  to  accomplish 
the  task,  the  ideas  behind  both  were  sound.  This  phase 
must  have  two  concurrent  efforts  (a  T&E  team  and  an 
acquisition  team)  to  take  the  synthesized  information 
from  the  “strategy-to-task”  phase  and  determine  T&E 
requirements  from  their  perspective.  These  requirements 
would  then  feed  into  the  next  phase,  “need-to-solution.” 

The  “need-to-solution”  phase  starts  with  the  input  of 
T&E  requirements  developed  by  the  T&E  and 
acquisition  communities.  These  requirements  are  the 
tasks  we  need  to  accomplish.  This  phase  will  determine 
how  we  solve  those  needs.  The  MSP  study  was  a  fairly 
successful  effort,  though  it  lacked  the  input  required  to 
make  it  a  fully  successful  plan.  The  synthesis  of 


information  from  the  “strategy-to-task”  phase  through  the 
“task-to-need”  phase  of  our  proposed  process  provides  a 
valuable  input  into  the  MSP.  The  process  for  development 
of  the  MSP  is  standardized  by  the  USAF,  but  the  quality 
of  the  product  is  not.  The  MSP  effort  would  interpret  the 
T&E  requirements  into  how  we  would  test,  what 
parameters  we  need  to  measiue,  etc,  and  determine  what 
capability  is  required  to  accomplish  the  test.  A 
comparison  with  existing  capabilities  would  define  the 
T&E  capability  deficiencies.  These  deficiencies  may 
include  upgrades  required,  sustainment  necessary,  or  new 
test  technology  required.  However,  up  to  this  point,  all  of 
the  information  provided  by  the  “strategy-to-task”  and 
“task-to-need”  phases  has  been  fiscally  unconstrained. 
This  means  the  tasks,  needs  and  solutions  defined  in  the 
T&E  MSP  were  also  unconstrained.  Thus,  the  next  step  in 
the  proposed  process,  the  Aimual  Plaiming  and 
Programming  Guidance  development.  This  is  an 
established  process  for  tire  USAF.  Joint  Vision  2010  states 
“As  we  implement  this  vision,  affordability  of  the 
technologies  envisioned  to  achieve  full  spectrum 
dominance  will  be  an  important  consideration.  While  we 
anticipate  that  some  significant  improvements  in 
capability  may  be  gained  economically,  others  will  be 
more  difficult  to  achieve  within  the  budget  realities  that 
exist  today  and  will  exist  into  the  next  century.  We 
anticipate  the  need  to  be  selective  in  the  technologies  we 
choose,  and  expect  continuing  assessment  and 
adjustments  for  affordability  as  well  as  for  other  lessons 
learned  during  the  implementation  process.”  The  APPG 
places  fiscal  reality  and  business  practices  constraints  in 
the  process.  These  constraints  provide  criteria  for 
determining  what  the  most  important  T&E  needs  are  that 
are  affordable.  By  following  these  constraints,  and 
improving  the  source  of  information  flowing  through  the 
MSP  into  the  APPG,  the  APPG  becomes  a  document  with 
increased  validity.  The  result  of  this  entire  effort  then  is 
identified,  affordable,  required  T&E  capabilities  and  T&E 
technology  needs  that  will  take  the  USAF  into  the 
21“  century. 

Summary 

The  USAF  has,  and  continues  to  reinvent  itself  since  the 
Cold  War.  All  commands  including  the  Air  Force  Material 
Command  have  gone  through  a  restructuring  to  account  for 
reductions  in  budgets  and  personnel.  As  part  of  Air  Force 
Material  Command,  the  T&E  community  also  is 
responding  to  change  by  evaluating  its  customers,  business 
practices  and  processes,  as  well  as  setting  strategic  goals. 
The  T&E  community  has  an  obligation  to  support  the 
USAF  vision  to  provide  “Global  Engagemenf’  in  support 
of  the  Joint  Chiefs  Joint  Vision  2010.  As  such,  flie  T&E 
long-range  modernization  vision  and  strategic  goals  are 
directly  linked  to  the  USAF  vision  for  the  21“  century.  This 
direct  linkage  ensures  T&E  modernization  planning  is 
consistent  wifli  USAF  guidance,  command  goals  and 
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objectives/plans.  Qjrrently,  the  USAF  T&E  community 
does  not  have  a  single  formalized  process  for  T&E 
capability  requirement  or  test  technology  needs 
identification  to  ensure  this  linkage  occurs. 

The  four  major  studies  discussed  in  this  paper  show  that 
the  USAF  has  been  working  diligently  for  the  past 
several  years  to  identify  T&E  capability  requirements  in 
preparation  for  the  2T'  century.  However,  as  can  be 
seen  from  these  discussions,  there  needs  to  be  a 
formalized  test  technology  needs  identification  process 
to  provide  for  integrating  existing  and  future  warfighter 
requirements  and  strategic  guidance  into  identifiable 
T&E  capability  requirements,  which  in  turn,  would 
allow  for  identification  and  discussion  of  new  test 
technology  needs  for  2T*  century  USAF  T&E.  Each  of 
the  discussed  studies  had  valuable  portions  of  each  of 
their  respective  processes. 

However,  the  U.S.  DoD  should  use  caution  when 
attempting  to  make  major  reduction,  restructuring, 
revitalization,  or  investment  decisions  based  on  the  data 
from  these  studies/plans.  With  the  limitations  identified 
in  each  study/plan  discussion,  as  well  as  the  probability 
that  there  will  be  programs,  either  classified  or  not  yet 
identified,  which  will  drive  significant  technological 
changes  to  our  required  tri-service  T&E  capabilities,  the 
chances  of  making  the  best  decision  based  on  this  data 
is  not  very  high.  These  studies/plans  provide  a 
tremendous  snapshot  and  analysis  of  T&E  requirements, 
but  because  the  requirements  are  based  primarily  on 
programmatic  documentation,  this  snapshot  is 
perishable  in  nature  and  must  be  routinely  updated.  In 
addition,  these  efforts  must  be  integrated  with  the 
futuristic  projections  as  mentioned  earlier  in  this  paper 
as  well  as  the  future  warfighter  requirements  in  order  to 
provide  a  clear  picture  that  will  allow  for  the  major 
T&E  reduction  and  investment  decisions  to  be  made 
with  a  high  level  of  confidence. 

From  past  experience  we  realize  that  the  type  of  efforts 
such  as  those  discussed  in  this  paper  will  recur  in  the 
near  future.  Rather  than  dropping  the  results  of  these 
efforts  and  starting  over  with  a  new  effort,  we 
recommend  that  the  USAF  T&E  community  use  the 
lessons  learned  from  accomplishing  these  studies  and 
develop  a  singular  formalized  process  such  as  that 
presented  in  this  paper.  This  singular  formalized  process 
should  provide  for  integrating  existing  and  future 
warfighter  requirements  and  strategic  guidance  into 
identifiable  T&E  capability  requirements,  which  in  turn. 


would  allow  for  identification  and  discussion  of  new 
test  technology  needs  for  2r'  century  USAF  T&E. 
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ABSTRACT 

The  capabilities  of  modeling  and  simulation 
(M&S)  have  grown  tremendously  during  the  past 
decade.  As  M&S  matures,  defense  officials  are 
starting  to  examine  the  possibility  of  using  this 
technology  to  improve  the  acquisition  and 
emplo)mt\ent  of  military  systems.  Potential 
applications  include  research,  development,  and 
engineering  of  new  systems;  test  and  evaluation; 
and  training.  Each  of  these  applications  has 
different  simxilation  requirements  that  must  be 
imderstood  before  simulation  development 
begins,  especially  since  the  distributed,  interactive, 
graphics-intensive  systems  typical  of  modem 
simulation  programs  have  historically  been 
developed  for  training  applications.  This  paper 
describes  the  general  requirements  that 
operational  testers  of  military  aircraft  will  place  on 
M&S  if  it  is  to  be  accepted  by  the  test  and 
evaluation  community.  Test  and  evaluation  -  and 
m  particular  operational  test  and  evaluation 
(OT&E)  -  may  pose  some  of  the  greatest 
challenges  to  M&S  developers,  but,  because  of  the 
limitations  inherent  in  the  large  field  tests 
characteristic  of  operational  testing,  OT&E  users 
could  benefit  greatly  from  modeling  and 
simulation  to  improve  their  test  programs.  More 
work  needs  to  be  done  in  M&S  development,  and 
the  challenges  facing  the  M&S  community  are 
discussed  in  this  paper. 

INTRODUCTION 

In  one  form  or  another  and  in  a  wide  variety  of 
applications,  modeling  and  simulation  (M&S)  has 
long  been  used  by  the  military.  Consider  the  first 
flight  simulators,  built  during  World  War  II  to 
train  pilots  quickly,  while  at  the  same  time 
reducing  the  amount  of  expensive  flight  training 
reqxoired  to  graduate  qualified  pilots.  Or  consider 
the  development  of  the  first  digital  computers, 
which  were  used  to  model  lethality  of  nuclear 
weapons  and  to  calculate  ballistic  missile 
trajectories. 

More  recently,  over  the  last  decade  or  so,  great 
strides  have  been  made  in  computer  networks  and 


graphics  capabilities,  pulled  along  by  the  ever- 
steady  rise  in  computational  power  and  speed  and 
mirrored  by  an  equally  rapid  decrease  in  cost  for 
these  capabilities.  At  the  same  time,  defense 
budgets  have  been  declining,  forcing  all  aspects  of 
military  operations  and  procurement  to  reduce 
costs.  To  alleviate  the  budgetary  presstue,  many 
military  departments  aroimd  the  world  have 
turned  to  M&S  as  a  means  to  do  more  with  less 
money,  and  whole  new  opportunities  for 
computer-based  M&S  have  arisen. 

Despite  the  potential  that  M&S  holds,  it  is  not  the 
panacea  for  every  situation.  Indeed,  it  is  only  a 
tool  that  should  be  used  judiciously. 

This  paper  discusses  the  use  of  M&S  in  the 
operational  test  and  evaluation  (OT&E)  of  aircraft 
-  the  promises  and  the  challenges.  OT&E  is  only 
one  part  of  the  acquisition  process,  but  it  is  one 
that  historically  has  not  enAusiasticaUy  embraced 
M&S.  Although  this  paper  focuses  on  OT&E  of 
aircraft,  the  observations  and  conclusions  in  this 
paper  should  apply  to  the  broader  topic  of  using 
M&S  in  cmy  test  application,  and  should  not 
necessarily  be  limited  to  military  programs. 

USER  DOMAINS 

One  of  the  first  things  to  recognize  is  tiiat  not  aU 
users  of  M&S  are  the  same.  Different  users  will 
generally  place  different  requirements  on  M&S, 
and  these  must  be  considered  carefully  when 
considering  the  use  of  M&S.  One  user  may 
require  a  man-in-the-loop  (MIL)  simulation  with 
high-resolution  visual  graphics  running  in  real 
time,  while  another  user  might  require  a  highly 
credible,  physics-based  model  that  does  not  have 
to  nm  in  real  time  and  does  not  require  a 
graphical  display.  In  other  words,  a  model  or  a 
simulation  developed  for  one  user  and  application 
may  not  be  suitable  for  another  user  with  a 
different  application. 

There  are  several  ways  to  delineate  mfiitary  user 
domains.  The  Simulation  Interoperability 
Standards  Organization  (SISO),  an  international 
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group  developing  M&S  standards/  has  defined 
five  categories  of  military  user  domains,  and  uses 
this  taxonomy  in  their  semi-annual  Simulation 
Interoperability  Workshop  (SIW): 

•  Analysis:  This  domain  uses  M&S  to  conduct  a 
wide  range  of  studies,  including  strategy, 
tactics,  and  policy  studies;  logistics  analyses; 
trade  and  early  concept  explorations  and 
feasibility  studies;  and  safety  investigations. 

•  Research,  development,  and  engineering 
(RD&E):  These  users  typically  employ  M&S 
early  in  the  acquisition  process  for  preliminary 
designs,  trade  studies.  Analysis  of 
Alternatives  (AOA)^  studies,  and 
manufacturing  feasibility  analyses. 

•  Small-unit  training:  This  domain  includes 
individual,  subteam,  and  team  training  to 
system  operators,  team  leaders,  tactical 
decision  makers,  and  other  hands-on  users. 

•  Staff-level  training:  This  domain  includes 
decision-makers  who  are  planning  and 
managing  the  activities  of  groups  of  people 
and  other  resources,  and  uses  M&S  for  the 
training  of  operational  and  strategic  leaders 
and  their  staff. 

•  Test  and  evaluation  (T&E):  These  users 
assess  system  performance  against 
requirements  or  specifications,  and  use  M&S 
later  in  the  acquisition  process,  typically  after 
Milestone  H,  the  entry  into  Engineering  and 
Manufacturing  Development. 

Taken  together,  these  domains  represent  the 
complete  cradle-to-grave  history  of  a  weapon 
system.  Diuing  concept  development,  analyses 
are  performed  to  determine  the  viability  and  cost 
effectiveness  of  various  approaches.  As  the  design 
matures  and  manufacturing  begins,  more  detailed 
RD&E  design  studies  are  performed.  Before  the 
system  is  fielded,  T&E  is  completed  to  ensure  that 
the  system  as  built  is  stiU  effective.  Once  the 
system  is  fielded,  operators  must  be  trained,  and 
further  analysis  is  done  to  determine  the  best  way 
to  employ  and  sustain  the  system. 

These  domains  are  roughly  independent,  although 
there  is  some  overlap.  For  example,  the  point  at 
which  analysis  ends  and  RD&E  takes  over  is  not 
distinct.  Likewise,  there  is  also  some  overlap  in 
the  requirements  that  each  domain  will  place  on 
M&S. 


’  See  <http://www.sisostds.org/>. 

^  This  was  formerly  called  the  Cost  and  Operational 
Effectiveness  Analysis  (COEA). 


It  is  important  to  understand  the  users  so  that 
simulations  can  be  tailored  to  meet  the  unique 
requirements  of  each  domain.  A  simulation 
developed  for,  say,  staff-level  training,  will  not 
necessarily  meet  the  needs  of  the  analysis 
community  without  modification. 

CASE  STUDY  OF  M&S  REQUIREMENTS: 
TRAINING  vs.  TEST  AND  EVALUATION 
To  illustrate  how  different  user  domains  might 
have  different  requirements  for  M&S,  we  will 
compare  some  general  M&S  requirements  of  two 
of  these  communities.  The  training  community  is 
a  good  starting  point  since  much  of  simulation's 
roots  is  in  the  training  community.  Although  the 
M&S  industry  is  trying  to  branch  out  to  other  user 
domains,  it  is  still  arguably  dominated  by  the 
military  training  commands.  Teaching  pilots  to  fly 
during  World  War  11  was  one  of  the  first  uses  of 
simulators  that  used  motion  and  other  cues  to 
improve  realism.  Flight  simulation  has  remained 
on  the  leading  edge  of  simulation  technology,  but 
other  training  applications  have  also  driven 
simulation  technology.  For  instance,  exercises  on 
a  scale  large  enough  to  train  staff-level 
commanders  are  too  expensive  actually  to  perform 
in  the  field,  but  they  can  be  simulated  through 
networked  computer  simulations,  and  have 
provided  some  of  the  motivation  for  linking 
distributed  simulations. 

The  primary  goal  in  training  is  to  transfer  positive 
skills  to  the  student.  Numerous  studies  have  been 
conducted  to  determine  what  qualities  of  training 
and  simulation  lead  to  optimum  transfer.  A 
review  of  these  studies  is  beyond  the  scope  of  this 
paper,  but  some  generalities  can  be  drawn 
concerning  the  use  of  simulators  in  training. 

Since  it  is  a  human  being  trained,  training 
applications  often  require  human  interaction  with 
the  simulation,  which,  in  turn,  dictates  that  most 
or  all  of  the  simulation  must  run  in  real  time. 

Many  MIL  simulations  also  require  high- 
resolution  visual  displays  to  recreate  faithfully 
what  the  trainee  would  see,  and  many  trainers  use 
force  and  audio  cues  to  increase  realism.  For 
many  training  applications,  it  is  important  to 
simulate  the  operational  procedures  accurately  so 
that  the  student  can  learn  the  proper  sequence  of 
events  to  perform.  However,  other  aspects  of  the 
simulation  do  not  necessarily  need  to  be  simulated 
accurately,  and  it  may  in  fact  be  beneficial  to  add 
some  artificiality.  For  example,  a  pilot  may  never 
experience  loss  of  an  aircraft  engine,  but  it  is 
obviously  helpful  to  practice  this  situation 
nximerous  times  in  a  flight  simulator.  In  general,  it 
is  not  an  a  priori  requirement  to  model  the  physics 
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of  the  processes  faithfully,  as  long  as  the  trainee  is 
immersed  in  an  adequately  realistic  environment 
that  leads  to  positive  skills  transfer. 

On  the  other  hand,  the  pmpose  of  T&E  is  to 
evaluate  certain  performance  characteristics  of  the 
system  vmder  test.  Since  T&E  is  fundamentally  a 
measurement  technique,  the  accuracy  of  the 
evaluation  is  an  overriding  concern. 

An  inaccurate  T&E  assessment  could  lead  to 
incorrect  production  decisions,  potentially  wasting 
acquisition  funds  and  fielding  inferior  or  imsafe 
systems.  As  desirable  as  it  is  to  achieve  high 
accuracy,  quantitatively  calculating  accuracy  for 
an  operational  test  program  can  be  difficult,  if  not 
impossible.  General  analytic  expressions  based  on 
measurement  theory  can  be  formulated,®  but  since 
these  rely  on  a  functional  description  between  the 
measurements  and  the  independent  variable  or 
subordinate  processes  (including  any  models), 
these  are  impractical  for  OT&E.  However,  an 
important  qualitative  result  from  measurement 
theory  is  that 

the  accuracy  of  the  final  measurement  is  in 
general  no  better  than  the  worst 
subordinate  error. 

What  does  this  have  to  do  with  modeling  and 
simulation?  If  we  view  M&S  as  a  tool  that  can  be 
used  to  evaluate  the  system  -  a  subordinate 
process,  in  effect  -  then  the  accuracy  of  the  overall 
evaluation  will  depend  on  the  accuracy  of  any 
model  or  simulation  used  in  a  test  program.  If  a 
model  or  a  simulation  with  poor  accuracy  or 
credibility  is  used  in  a  key  part  of  an  evaluation,  it 
will  inordinately  degrade  the  overall  accuracy  of 
the  final  evaluation.  Of  coxurse,  this  doesn't  just 
apply  to  simulation,  but  applies  to  the  entire 
OT&E  program. 

OPERATIONAL  TEST  AND  EVALUATION 
Two  laws^  were  passed  in  the  mid-1980s  to  create 
the  office  of  the  Etirector,  Operational  Test  and 
Evaluation  (DOT&E)  and  to  formalize  the  role  of 
OT&E  in  the  acquisition  process.  These  laws  state 
that: 

a  major  defense  acquisition  program  may 
not  proceed  beyond  low-rate  initial 
production  until  initial  operational  test 


®  "Accuracy  in  DoD  High  Level  Architecture 
Federations,"  Lester  Forster,  paper  97F-SIW-064 
presented  at  the  Fall  1997  Simulation 
Interoperability  Workshop  (see  footnote  1). 

"  10  use  §139  and  10  use  §2399. 


and  evaluation  of  the  program  is 
completed. 

Congress  defined  operational  test  and  evaluation 
as  follows: 

"operational  test  and  evaluation"  means  - 
the  field  test,  imder  realistic  combat 
conditions  and  may  not  be ...  based 
exclusively  on  computer  modeling  [or] 
simulation 

These  laws  force  major  military  systems  to  be 
tested  by  an  independent  agency  in  realistic, 
warlike  situations  with  real  operators.  They  are 
intended  to  prevent  purchasing  systems  justified 
solely  by  paper  or  computer  studies,  or  only  by 
sterile,  laboratory  tests.  By  placing  systems  in 
stressful  situations  before  production  begins,  flaws 
can  be  discovered  and  corrected  before  the  system 
is  bought  and  fielded.  If  the  system  had  too  many 
flaws,  or  if  the  system  met  all  of  its  technical 
specifications  yet  was  stiU  not  effective  in  the  field, 
the  program  could  be  terminated  before  investing 
large  amoimts  of  money  in  full-rate  production. 
There  is  also  the  perception  that  field  tests  are 
more  honest  and  less  susceptible  to  political 
manipulation  than  models  or  simulations.  Thus, 
the  operational  evaluation  provided  by  DOT&E 
allows  the  Milestone  IH  Full  Rate  Production 
decision  to  be  made  with  higher  confidence  and 
with  less  risk. 

Although  the  law  was  intended  to  emphasize 
field-testing.  Congress  clearly  allows  DOT&E  to 
include  some  modeling  and  simulation  in  its 
evaluations.  Congress  recognized  that  field 
testing  by  itself  cannot  permit  a  complete  and 
perfectly  accurate  evaluation,  and  that  there  is  a 
role  for  analysis  above  and  beyond  the  results 
obtained  in  the  field. 

Recall  earlier  that  it  was  asserted  that  since  OT&E 
essentially  measures  the  performance  of  a  system, 
then  it  is  of  paramoxmt  importance  that  this 
measurement  be  accurate.  For  OT&E,  accuracy  is 
defined  as: 

obtaining  an  evaluation  of  performance 
from  the  test  program  that  is  close  to  the 
level  of  performance  that  would  be 
obtained  in  an  actual  wartime  situation. 

According  to  this  definition,  the  most  accurate 
method  of  measuring  performance  of  a  military 
system  is  to  place  it  in  an  actual  war  (and  indeed, 
this  is  why  Congress  created  the  operational-test 
laws  described  above),  but,  of  course,  this  is  not 
practical.  Field  tests  can  be  made  to  approximate 
wartime  scenarios,  but  the  maxim  that  "anything 
short  of  war  is  a  simulation"  is  true. 
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MEASURES  OF  EFFECTIVENESS  FOR 
AIRCRAFT  OT&E 

Besides  the  test  environment  itself,  the  measures 
used  to  characterize  system  performance  also  play 
a  key  role  in  developing  the  operational  test 
program.  For  OT&E,  these  measures  of 
effectiveness  (MOEs)  are  divided  into  two 
categories  -  operational  effectiveness  and 
operational  suitability.  The  former  describes  how 
well  the  system  performs  its  intended  mission, 
while  the  latter  characterizes  the  ability  to  which 
the  system  can  be  used  in  its  intended  application. 

MOEs  used  in  OT&E  are  generally  at  the  mission 
level  (i.e.,  they  describe  the  outcome  of  a  mission). 
They  can  also  be  at  a  higher,  campaign  level,  but 
MOEs  at  the  lower,  engagement  are  also  used  in 
OT&E.  There  is  no  established  guidance  that 
dictates  the  level  that  should  be  used,  and,  indeed, 
there  may  be  inconsistency  among  programs  and 
services.  One  factor  in  determining  the  most 
appropriate  level  is  how  the  Operational 
Requirements  Document  (ORD)®  specifies 
requirements  for  the  system.  Another  important 
consideration  is  the  AOA:  since  the  AOA  justifies 
the  start  of  procurement,  it  is  logical  to  use  the  test 
program  to  confirm  the  predictions  made  in  this 
study. 

Measures  of  operational  effectiveness  depend  on 
the  role  of  the  system  being  tested.  For  fighter 
aircraft  that  have  the  role  of  destro3dng  enemy 
aircraft,  an  effectiveness  MOE  might  be  the  loss- 
exchange  ratio,  which  expresses  the  ratio  of  enemy 
and  friendly  aircraft  killed  in  a  specified  time, 
such  as  over  a  mission  or  over  a  campaign.  For 
strike  aircraft,  an  appropriate  measure  could  be 
the  ability  to  penetrate  enemy  air  defenses  and 
deliver  the  ordnance  accurately  to  the  target. 
Useful  measures  for  surveillance  aircraft  might 
include  detection  range,  or  the  number  or 
percentage  of  targets  detected.  Measures  that 
have  been  used  to  evaluate  transport  aircraft 
include  miUion-ton-miles  per  day  (MTM/D)  of 
cargo  transported,  and  closure  time  to  airlift  a 
military  unit.  These  measures  are  summarized  in 
Table  1. 

Besides  these  effectiveness  measures,  every 
aircraft  procvued  has  operational  suitability 
requirements  that  it  must  meet.  The  requirements 
that  usually  receive  the  most  attention  are  those 
concerned  with  reliability,  maintainability,  and 
availability  (RM&A).  At  the  highest  level,  most 
aircraft  will  be  required  to  maintain  a  sustained. 


*  The  ORD  specifies  what  the  user  of  the  system 
requires  of  the  system. 


high-tempo  level  of  operations  that  might  be 
expected  in  wartime.  This  capability  can  be 
measured  in  terms  of  sorties  per  day  (typical  of 
fighters),  utilization  rate  (transports),  or  on-station 
availability  (surveillance  aircraft).  Other 
requirements  may  specify  the  maintenance 
demands  and  the  reliability  of  the  aircraft  and  its 
systems.  Examples  of  these  include  maintenance 
man-hotus  per  flight  hour  (MMH/FH),  mean 
repair  time,  and  number  of  maintenance  actions 
per  flight  hour. 

Besides  RM&A  requirements,  there  are  often  other 
important  suitability  requirements.  These  might 
include  interoperability  between  other  similar 
assets  (increasingly  important  as  newer  systems 
are  fielded  with  more  command  and  control 
capability  and  are  expected  to  participate  in  such 
networks),  interoperability  with  other  service's 
platforms  and  systems,  and  logistics  support 
including  spares  provisioning,  training  equipment 
and  planning,  and  depot  repair  support. 

Some  of  the  highest-level  measures  can  combine 
both  effectiveness  and  suitability  measures.  These 
are  usually  campaign-level  measures,  which  might 
include  the  same  effectiveness  measures  described 
above  but  -  rather  than  for  just  one  mission  or 
sortie  -  for  an  extended  period  of  time.  For 
example,  for  a  bomber  aircraft,  the  relevant 
campaign-level  measure  might  be  the  number  of 
targets  destroyed  diuing  a  month-long  campaign. 
Sud\  meastues  are  attractive  for  several  reasons. 
One  is  that  they  combine  both  operational 
effectiveness  and  suitability  measrues.  Another  is 
that  measures  of  this  level  are  often  used  to  justify 
procurement  in  the  AOA.  Finally,  measures  at 
this  level  are  an  intuitive  way  of  communicating 
overall  performance  to  decision-makers. 

THE  PROMISE  FOR  MODELING  AND 
SIMULATION 

Now  that  we  have  defined  the  desired  operational 
test  environment  and  the  performance  parameters 
we  need  to  obtain  from  the  test  program,  we  can 
discuss  our  ability  to  achieve  these  goals.  If  we  are 
unable  to  measvue  all  of  the  desired  MOEs 
completely  in  the  stressing  wartime  environment 
through  fUght  testing,  M&S  may  be  able  to  fill  in 
some  gaps  and  improve  the  operational 
assessment. 

Constructing  realistic  operational  tests  might 
soimd  easy,  but  when  the  innumerable  details  are 
considered,  it  becomes  apparent  that  creating  a 
highly  realistic  test  environment  can  be 
challenging,  especially  for  aircraft.  Some  of  the 
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Table  1.  A  Sampling  of  Engagement  and  Mission-Level  Effectiveness  MOEs 
for  Tactical  Aircraft  Programs 


Mission 

Aircraft 

Mission-Level  Measures 

Strike 

F/A-18E/F 

JSF 

Number  of  strike  aircraft  reaching  weapons  reiease  point 

Number  of  sorties  required  to  destroy  a  specified  target 

Number  of  aircraft  lost  destroying  a  specified  target 

Air  superiority 

F-22 

F/A-18E/F 

JSF 

Exchange  ratio  (defined  as  the  number  of  enemy  aircraft  lost  divided 
by  the  number  of  friendly  aircraft  lost) 

Dominance  (defined  as  the  number  of  trials  in  which  50%  of  the 
opponents  were  killed) 

Percent  of  trials  In  which  high-value  airborne  assets  are  killed 

Strategic 

transport 

C-17 

Million-ton-mile/day  (MTM/D)  capacity 

Throughput  vs.  airfield  size,  tons/day 

Time  and  sorties  required  to  deploy  a  specified  unit 

Tactical 

C-17 

Throughput  vs.  airfield  size,  tons/day 

transport 

C-130J 

V-22 

Time  and  sorties  required  to  deploy  an  airborne  unit 

Tons/day  deliverable  vs.  sortie  radius 

Time  required  to  airlift  an  air  assault  element  of  a  marine 
expeditionary  brigade  ashore  (V-22) 

The  reduction  in  time  (compared  to  existing  systems)  required  to 
evacuate  50  casualties  from  a  combat  zone  (V-22) 

Probability  of  successfully  completing  a  special  operations  infiltration 
mission  in  a  moderate  air-defense  environment 

Surveillance 

JSTARS 

AWACS 

Timeliness,  accuracy,  and  completion  of  delivering  detection  reports 
(JSTARS) 

Fraction  of  hostile  aircraft  detected  before  weapons  release 
(AWACS) 

limitations  encountered  in  the  operational  test  of 
aircraft  include: 

•  Threat  replication:  Right  testing  against  actual 
enemy  systems  (including  aircraft,  missiles, 
radars,  and  infrastructure),  operated  by  enemy 
crews,  does  not  occur.  Exploitation  of  foreign 
systems,  when  it  does  occur,  is  usually  limited 
to  only  a  handful  of  systems,  insufficient  to 
conduct  a  meaningful  operational  test. 

•  Force  density:  There  are  many  systems  that 
might  be  present  and  relevant  during  an  actual 
air-combat  sortie:  escort  aircraft,  strike  aircraft, 
jammers,  tankers,  surveillance  aircraft,  groimd 
radars,  and  command-and-control  networks.  It 
is  costly  and  complex  to  have  all  relevant 
systems  participate  in  a  flight  test. 

•  Kill  assessment  and  removal:  In  a  flight  test, 
no  one  actually  dies,  and  systems  are  not 
usually  destroyed.  Thus,  something  that  was 
supposedly  killed  in  a  test  wfil  stfil  exist  and 
may  remain  as  a  shadow  factor  in  the  mind  of 
the  operator  or  in  the  system  itself  (e.g.,  radars 
or  other  sensors).  Such  residual  presence  must 
be  artificially  overruled. 


•  Safety:  Missiles  and  guns  cannot  be  fired  at 
aircraft,  even  if  the  missiles  contain  inert 
warheads.  Night  and  low-level  fl)dng  is  also 
usually  restricted. 

•  Range  space:  Instrumented  range  space  is 
limited  in  the  United  States,  which  could 
impose  an  artificial  barrier  to  systems  imder 
test.  Fighter  and  strike  aircraft  are  usually 
capable  of  supersonic  speeds,  which  in  turn 
require  more  expansive  ranges.  Pilots  must 
always  be  aware  of  where  they  are  in  relation 
to  the  range  boimdaries. 

•  Expense:  Fli^t  testing  is  expensive.  For  even 
a  modest  few-on-few  engagement,  20  or  more 
aircraft  could  be  required  to  be  airborne  when 
aU  support  aircraft  are  considered.  Wartime 
conditions  imply  a  high  tempo  of  operations  to 
assess  operational  suitability.  Multiple  trials, 
consisting  of  repeated  runs  for  statistical 
cojrfidence  or  for  exploring  different  scenarios, 
are  desirable,  further  compounding  the 
expense.  Maintenance,  logistics,  and  support 
costs  can  also  be  considerable. 
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Because  of  these  limitations,  direct  flight  testing 
carmot  directly  assess  all  of  the  MOEs  typically 
desired  from  OT&E  of  aircraft.  Each  of  the 
limitations  described  above  presents  an 
opportunity  for  M&S  to  improve  the  operational 
evaluation  of  the  system. 

Consider  the  first  two  limitations  described  above: 
threat  replication  and  force  density.  Simulations 
running  concurrent  with  the  test  can  simulate 
many  of  these  features.  A  threat  can  be  replicated 
artificially  by  an  off-board  simulation  and  injected 
into  the  aircraft;  if  this  stimulation  occurs  just  after 
the  sensor,  not  only  can  the  pilot  be  led  to  believe 
that  the  threat  is  present,  but  the  signal-processing 
algorithms  of  the  aircraft  can  also  be  evaluated  to 
enstue  that  they  properly  process  the  threat.  If 
enough  threats  are  simulated  in  this  maimer,  the 
threat  density  can  be  increased  to  simulate  a 
larger,  stressing  engagement.  The  ability  of  both 
the  operator  and  the  sensor-fusion  algorithms  to 
process  large  amounts  of  information  is  becoming 
more  important  in  modem  aircraft,  and  thus 
simulation  could  play  a  large  part  in  evaluating 
this  capability. 

The  next  two  considerations  are  kiU  assessment 
and  removal,  and  safety  limitations  preventing  the 
firing  of  missiles  and  gims.  Both  of  these 
limitations  can  be  somewhat  mitigated  with 
simulation.  Missile  flyouts  can  be  simulated,  and 
a  miss  or  kill  can  be  based  on  a  random  draw  if  the 
inissile  gets  close  enough  to  the  target  to  fuse.  If 
the  target  is  killed,  simulation  can  also  be  used  to 
artificially  suppress  the  display  of  that  target  on 
the  other  pilot's  displays,  hideed,  if  the  air- 
combat  battle  remains  beyond-visual  range  (BVR), 
the  only  knowledge  a  pilot  has  of  enemy  aircraft  is 
through  his  display;  if  this  is  controlled 
completely  by  &e  simulation,  then  theoretically 
the  information  presented  to  the  pilot  in  this  test- 
simulation  hybrid  would  be  the  same  as  an  actual 
wartime  engagement. 

Adequate  test  range  space  is  becoming  an 
increasing  problem  as  aircraft,  such  as  the  F-22, 
acquire  supercruise  capability.  Tests  conducted  at 
supercruise  conditions  exceeding  Mach  1.4  for 
periods  exceeding  15  minutes  (routinely  practical 
without  afterburners)  will  cause  the  aircraft  to 
exceed  the  existing  test  range  botmdaries  of 
Edwards,  Nellis,  and  Eglin  Air  Force  Bases. 
Although  it  has  been  proposed  to  link  the  ranges 
at  these  Air  Force  bases  at  high  altitudes  (above 
commercial  airliner  altitudes)  to  support  these 
supercruise  tests,  this  proposal  has  not  been 
approved  (and  likely  will  never  be  approved)  by 
the  Federal  Aviation  Authority. 


The  final  limitation  listed  above  is  cost.  The 
potential  for  M&S  to  alleviate  some  of  the  high 
cost  of  testing  is  discussed  in  more  detail  later  in 
this  paper. 

Notice  that,  in  the  examples  cited  in  this  section, 
M&S  is  integrated  into  the  flight-test  program. 

This  is  an  important  observation,  since  too  often 
planners  approach  flight  testing  and  M&S  as 
opposites.  ITie  leads  to  a  partial  decoupling  of 
flight  testing  and  M&S,  and  may  result  in  a  test 
program  that  consists  of  distinct  modeling  and  test 
phases,  e.g.,  the  "model-test-model"  approach. 
There  is  nothing  inherently  wrong  with  this 
approach,  but  it  can  relegate  the  role  of  testing  to 
merely  providing  data  for  model  validation,  and  it 
does  not  take  advantage  of  the  possible  synergism 
between  modeling  and  simulation.  This 
synergism  was  illustrated  in  the  examples  dted 
above. 

There  are,  of  course,  times  in  OT&E  where  the  use 
of  M&S  is  not  necessarily  integrated  into  flight 
testing.  In  a  previous  paper,*  we  described  using 
M&S  to  extrapolate  test  data  both  horizontally  (at 
the  same  level  of  MOE),  and  vertically  (to  higher 
level  MOEs)  after  flight  testing  was  completed. 
This  is  often  required  to  expand  the  range  of  the 
results  -  e.g.,  to  threats  or  scenarios  that  could  not 
be  tested  -  or  to  address  performance  at  a  level 
that  could  not  be  tested  -  e.g.,  at  a  campaign  level. 

Modeling  and  simulation  can  also  improve  OT&E 
by  being  used  in  pre-test  activities,  including  the 
following: 

•  Rehearsal:  Flight  testing  is  expensive.  By 
rehearsing  the  test  in  a  simulator,  the  chance 
that  a  test  wiU  be  ruined  or  compromised 
because  an  operator  did  not  fulM  his  role  in 
the  scenario  can  be  reduced. 

•  Scenario  selection:  Usually  there  are  more 
scenarios  of  interest  than  can  be  actually  flight 
tested.  By  analyzing  the  scenarios  with  M&S, 
the  best  scenarios  can  be  selected  for  flight 
testing.  These  scenarios  can  be  chosen  by 
selecting  the  most  stressing  cases,  or  they  can 
be  chosen  by  selecting  the  operationally 
representative  cases,  or  by  other  criteria. 

•  Training:  Like  rehearsal,  expensive  flight 
time  can  be  reduced  by  training  operators  in 
simulators  before  flight  testing.  Consistent 
levels  of  maturity  -  useful  for  ensuring 
comparability  of  test  results  -  between 


*  "Modeling  and  Simulation  in  Operational  Test," 
David  Wiesenhahn  and  Robert  Dighton,  presented 
at  the  1995  Military,  Government  and  Aerospace 
Simulation  MultiConference,  April  1995. 
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operators  can  also  be  achieved  via  a 
simulation-based  training  program. 

•  Tactics  development:  Tactics  usually  take  a 
while  to  develop  for  new  systems,  yet  tactics 
can  influence  the  effectiveness  of  the  system. 
Pre-test  simulation  can  be  used  to  develop  and 
refine  tactics  that  make  optimum  use  of  the 
new  system's  capabilities  before  testing. 

THE  CHALLENGES  FOR  MODELING  AND 
SIMULATION 

Accuracy 

The  previous  section  described  various  ways  in 
which  M&S  could  be  used  in  an  aircraft  OT&E 
program.  It  was  suggested  that  there  is  potential 
for  integrating  M&S  into  the  flight-testing 
environment,  and  possible  pre-  and  post-test  uses 
for  M&S  were  also  discussed. 

Throughout  the  test-planning  process,  the  goal  of 
achieving  an  accurate  evaluation  must  always  be 
kept  foremost  in  mind,  especially  when 
incorporating  M&S  into  the  test  program  is 
considered.  It  makes  no  sense  to  attempt  to 
correct  a  shortcoming  in  the  test  program  with 
M&S  if  the  M&S  actually  degrades  the  accuracy  of 
the  result. 

In  listing  the  uses  for  M&S  above,  a  distinction  can 
be  made  between  those  that  directly  influence  the 
accuracy  of  the  results  and  those  that  have  less 
influence;  accordingly,  the  acciuacy  demands  for 
M&S  used  in  the  latter  group  will  be  less.  M&S 
used  in  pre-test  activities  will  generally  have  less 
influence  on  the  accuracy  of  the  evaluation.  For 
example,  if  the  flight-test  scenarios  were  chosen 
based  on  an  M&S  analysis  and  these  turned  out 
not  to  be  the  best  choices  of  scenario  (by  whatever 
criteria  were  used),  the  evaluation  results  can  stiU 
be  accurate,  even  if  they  do  not  necessarily  apply 
to  the  optimum  scenarios. 

On  the  other  hand,  M&S  used  directly  in  flight 
testing  and  for  post-test  analysis  and  extrapolation 
will  usually  have  a  direct  influence  on  accuracy. 

To  take  a  simple  example  discussed  above,  it  is 
impossible  actually  to  laimch  missiles  at  piloted 
aircraft.  This  can  be  corrected  by  using  a  model  to 
simulate  the  missile  laimch,  guidance,  and  fuzing. 
However,  if  for  whatever  reason  the  simulation  is 
not  sufficiently  accurate  or,  in  the  worst  case, 
contains  a  bias  toward  one  system,  misleading 
results  could  occur.  In  this  case,  it  may  be  better 
to  back  up  a  step,  and  examine,  for  example,  the 
possibility  of  using  shot  opportunities  (i.e.  before 
missile  launch)  as  an  MOE.  Indeed,  as  discussed 
above,  the  inaccuracy  induced  by  this  simulation 
could  dominate  smaller  inaccuracies  throughout 


the  remainder  of  the  test,  which  may  have  been 
achieved  only  through  costly  and  painstaking 
effort.  Thus,  in  this  example,  even  though  a 
deficiency  in  the  test  program  had  supposedly 
been  corrected  by  M&S,  it  is  possible  to  wind  up 
with  a  less  desirable  evaluation. 

Even  though  accuracy  is  important  to  aU  users 
(not  just  test  and  evaluation  users),  few  models 
and  almost  no  simulations  report  what  kind  of 
accuracy  can  be  expected  when  using  these  tools. 
Sometime  models  and  simulations  are  given 
caveats  that  they  should  be  used  only 
qualitatively,  e.g.,  to  determine  trends  or  to  rank 
alternatives.  Part  of  most  verification,  validation, 
and  accreditation  (W&A)  processes  is  a 
determination  of  the  limits  on  where  the  model 
can  be  used  and  remain  accredited;  this  is 
presumably  based  on  some  predefined  accuracy, 
but  for  the  most  part,  the  W&A  process  does  not 
explicitly  yield  an  estimate  of  accmacy. 

Altiiough  the  definition  of  the  accuracy  of  a  model 
or  simulation  is  conceptually  easy  to  imderstand  it 
can  still  be  difficult  to  quantify  accuracy  for  a 
simulation  that  models  something  as  complex  as  a 
military  battle.  Clearly,  accuracy  should  be  based 
on  how  well  the  output  from  the  model  or 
simulation  agrees  with  reality,  but  at  what  level 
should  this  comparison  be  made?  Consider  the 
missile  fly-out  model  discussed  above.  A 
comparison  could  be  made  based  on  the  overall 
probability  of  loll  (Pk)  given  any  shot  within  the 
weapon-engagement  zone.  It  could  also  be  based 
on  Pk  segregated  by  launch  conditions.  The 
comparison  could  also  be  made  at  a  lower  level, 
such  as  miss  distance  or  fuzing  distance  and 
geometry.  At  an  even  lower  level,  the  signals 
generated  and  received  by  tiie  actual  seeker  could 
be  compared  to  those  in  ^e  model,  as  coiild  the 
fin  deflection  and  guidance  commands.^ 

All  models  and  simulations  -  even  physics-based 
models  -  will  have  some  compromises  and 
limitations  known  to  the  devdopers.  If  the 
accuracy  of  the  model  or  simulation  carmot  be 
quantified  in  a  rigorous  manner,  the  limitations 
should  at  least  be  disdosed  as  part  of  the 
program's  documentation  so  that  the  user  can 
determine  if  it  meets  his  needs.  Convincing  the 
developers  actually  to  do  this  may  be  difficult, 
because  they  might  fed  that  a  limitation  in  their 
model  could  be  perceived  as  a  defidency. 


’’  This  approach  is  used  by  the  Joint  Accreditation 
Support  Activity  (see 

<http://www.nawcwpns.navy.mil/~jasa/ >. 
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At  the  beginning  of  this  paper,  we  stated  that 
different  users  will  generally  have  different 
modeling  and  simulation  requirements,  likewise, 
different  users  will  have  different  ways  to  describe 
the  quality  of  a  model  or  simulation.  A  common 
term  used  to  describe  this  quality  is  "fidelity." 
Unfortunately,  there  is  no  standard  definition  of 
fidelity,  and  this  term  may  mean  different  things 
to  different  users.  Other  terms,  such  as 
"credibility,"  "resolution,"  and  "precision"  are 
related  to  fidelity  and  are  often  used  to  describe 
M&S  quality,  but  these  terms  also  lack  an  accepted 
set  of  rigorous  definitions. 

One  use  of  "fidelity"  is  based  on  the  resolution  of 
displays,  input  data,  or  time  step  used  in  the 
simulation.  In  this  case,  the  higher  the  resolution, 
the  higher  the  fidelity.  To  some  user  communities, 
such  as  training,  this  definition  is  important  and  a 
valid  measure  of  simulation  quality. 

Another  common  usage  of  "fidelity"  is  that  it 
describes  the  agreement  between  die  model  or 
simulation  and  the  physical  world.  This  can  be 
based  on  comparing  the  number  of  entities  in  the 
simxalation  to  those  in  the  battle  being  simulated, 
and  leads  to  the  common  notion  that  the  bigger 
the  simulation  is,  the  better  it  is.  However,  if  the 
comparison  between  reality  and  simulation  is 
based  on  the  output  of  the  simulation,  then 
"fidelity"  corresponds  to  "accuracy"  as  we  have 
been  using  it  in  this  paper.  Note  that  simply 
increasing  the  resolution  or  the  number  of  entities  in 
the  simulation  does  not  necessarily  guarantee  increased 
accuracy. 

Care  must  be  taken  when  discussing  the  quality  or 
fidelity  of  a  model  or  simulation  to  ensure  that  the 
definition  is  xmderstood  by  the  user  and  it  is 
appropriate  for  the  user's  application. 

Simulating  Complex  Warfare 

Given  the  emphasis  on  accuracy  throughout  this 
paper,  it  is  a  fair  question  to  ask:  are  we  capable  of 
accurately  simulating  large-scale  warfare  on 
digital  computers?  Even  if  we  had  infinite 
computational  resources,  can  we  write  algorithms 
for  all  the  physical  processes  relevant  to  the  battle, 
and  can  we  populate  those  algorithms  with 
sufficiently  accurate  input  data?  Can  we 
characterize  the  human  element  -  so  important  in 
all  warfare  -  in  terms  that  a  computer  can  process? 
Can  we  even  identify  all  of  the  factors  a  priori  that 
influence  the  outcome  of  a  battle? 

Military  warfare  includes  a  tremendous  range  of 
physical  phenomena,  and,  to  model  these,  we 
must  be  able  to  describe  each  phenomenon 
analytically  in  madiematical  terms.  For  example. 


air  combat  involves  propulsion  (from  jet  engines 
and  missile  motors);  aerodynamics  (for  both 
aircraft  and  missiles);  electromagnetic  emission, 
propagation,  and  interaction  (radar  and  infrared 
sensors);  signal  processing  (radars,  sensors,  and 
avionics);  and  ballistics  (warhead  detonation  and 
effects).  Environmental  phenomena,  such  as 
weather  and  backgroimd  clutter,  must  also  be 
considered.  Fully  describing  each  of  these 
phenomena  with  computer  algorithms  that 
minimize  error  are  fuU-fledged  research  areas  in 
their  own  right,  and  bringing  all  of  these  together 
in  a  military  simulation  may  be  an  impossible 
goal. 

Besides  modeling  the  physical  processes 
themselves,  the  interaction  between  these 
processes  must  also  be  modeled.  Sensors  interact 
with  man-made  systems  (both  friendly  and 
enemy)  and  with  the  environment.  Human 
operators  interact  with  displays  and  flight 
controls.  Missiles  receive  cues  from  the  host 
system  before  laimch,  then  interact 
aerodynamically  with  the  atmosphere.  During 
guidance,  the  missile  may  continue  to  interact 
with  the  host  system  or  it  will  interact  with  the 
target  aircraft,  which  may  be  maneuvering  or 
changing  its  signature.  If  the  missile  fuzes  and 
detonates,  the  warhead  fragments  will  interact 
with  the  targeted  aircraft. 

Even  if  algorithms  are  developed  for  all  the 
physical  processes  and  their  interactions,  some 
amount  of  input  data  will  be  required.  Ensuring 
the  quality  of  these  input  data  is  important,  since 
the  overall  accuracy  of  the  model  is  directly 
correlated  to  the  accuracy  of  the  input  data.  These 
input  data  could  be  fundamental  physical- 
property  data  -  for  example  the  temperature 
distribution  and  emissivity  of  the  airplane  skin, 
required  to  calculate  its  infrared  (IR)  signature. 
These  input  data  could  also  be  measurements 
made  in  developmental  test,  such  as  the  radar 
cross-section  (RCS)  of  an  aircraft.  Again, 
considering  the  breadth  of  a  force-on-force 
military  simulation,  these  data  requirements  may 
be  significant.  For  example,  in  order  to  accurately 
calculate  the  IR  signature,  the  emissivity  of  the 
aircraft  skin  in  the  above  example  should  be 
known  as  a  function  of  angle,  wavelength,  and 
temperature,  and  the  temperature  would  need  to 
be  known  as  a  function  of  the  flight  envelope.  In 
the  other  example,  the  RCS  may  need  to  be  known 
as  a  function  of  frequency,  and  at  closely  spaced 
values  of  azimuth,  and  elevation.  This  volume  of 
data  may  be  onerous  for  friendly  systems,  and 
completely  impractical  to  obtain  for  enemy 
systems. 
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Authoritatively  modeling  human  behavior  is 
difficult,  yet  it  is  an  important  factor  m  combat. 
The  outcome  of  a  battle  can  be  decisively 
influenced  by  strategic  decision-makmg,  tactical 
action  and  reaction,  human  blimder,  or 
imcommon  valor.  Some  air-combat  models  have 
attempted  to  model  the  human  decision-making 
process.  Ho^vever,  W&A  of  the  human  part  of 
these  models  has  not  been  done  rigorously,  and, 
indeed,  a  rigorous  method  of  validating  human 
behavior  may  not  be  possible.  One  option  is  to 
obviate  the  need  for  digital  human  models  by 
using  man-in-the-loop  simulators.  Of  course, 
special  effort  may  be  reqxiired  to  train  some 
participants  to  act  like  foreign  operators. 

Another  challenge  facing  simulator  developers  is 
to  model  "unknown-unknowns."  A  simulation  or 
model  can  only  do  what  it's  been  programmed  to 
do,  yet  m  every  test,  and  in  warfare  itself,  things 
that  were  completely  imanticipated  will  occur. 
Despite  the  system  designer's  best  efforts  to 
anticipate  every  use,  misuse,  and  problem  that 
could  occur  with  his  system,  these  imanticipated 
unknown-unknovms  may  be  serious  deficiencies 
with  the  system,  and  uncovering  these  problems 
may  be  an  extremely  important  test  result. 

Cost 

Simulations  are  often  touted  as  being  far  cheaper 
than  field  exercises.  This  claim  is  usually  made  on 
the  basis  of  the  cost  to  run  one  engagement  of  a 
computer  simulation  compared  to  the  cost  of 
performing  one  similar  exercise  in  the  field  witti 
real  troops  and  hardware.  On  this  basis,  the  field 
test  might  cost  tens  or  himdreds  of  thousands  of 
dollars,  or  even  millions  of  dollars,  whereas  an 
execution  of  the  simulation  might  be  many  orders 
of  magnitudes  less. 

Considering  the  deployment  and  sustenance  of 
personnel  and  equipment  sufficient  to  replicate  an 
engagement,  it  is  true  that  field  testing  can  be 
quite  expensive,  but  it  has  not  yet  been 
demonstrated  that  simulation  offers  aU  of  the  cost 
savings  anticipated.  One  problem  with  comparing 
the  cost  of  simulation  to  field  testing  is  that  when 
the  cost  of  simulation  is  reported,  it  is  usually  the 
cost  required  to  run  one  scenario,  and  does  not 
include  the  capital  cost  required  to  develop  the 
simulation,  nor  does  it  include  the  cost  to  maintain 
the  code.  W&A  costs  can  be  an  alarmingly  high 
part  (at  least  to  the  simulation  developer)  of  the 
total  simulation  development  cost,  but  are  also 
usually  not  included  in  the  price  quoted  to  run  the 
model. 


Tied  in  with  this  discussion  is  the  notion  that  since 
simulations  are  cheap  to  run,  they  can  be  executed 
multiple  times  to  gather  "data,"  improving  the 
statistical  confidence  in  the  results  and  precluding 
the  need  to  conduct  multiple  trials  in  the  field.  For 
this  to  be  true,  the  variations  in  the  fundamental 
processes  that  cause  variability  in  the  actual  tests 
must  be  characterized  and  replicated  in  the 
simulation.  However,  if  we  can  conduct  enough 
tests  to  characterize  these  processes,  it  may  be 
argued  that  we  have  conducted  enough  tests  to 
characterize  the  system,  and  that  additional 
simulation  runs  will  not  provide  any  useful 
information.  Thus,  this  is  a  circular  argument. 
There  is  an  exception  to  this,  which  presents  a 
good  opportunity  for  M&S.  If  components  of  the 
actual  system  (including  the  human  component) 
are  embedded  into  the  simulation,  a  priori 
knowledge  of  their  statistical  behavior  will  not  be 
required. 

Few  studies  have  been  done  to  determine  the  cost 
savings  obtained  by  using  M&S  in  test  and 
evaluation  in  particular  and  acquisition  in 
general.®  There  is  no  doubt  that  engineers  will 
use  M&S  when  it  intuitively  makes  sense,  and 
there  is  no  doubt  that  M&S  has  improved  product 
quality,  decreased  development  time,  and  lowered 
the  cost  in  both  the  commercial  and  military 
sectors.  However  to  truly  quantify  the  value  that 
M&S  adds  to  military  acquisition,  rigorous  and 
objective  studies  must  be  performed,  preferably 
based  on  a  comparative  analysis;  however,  given 
the  uniqueness  of  large  military  acquisition 
programs,  this  will  be  difficult. 

CONCLUSIONS 

Operational  test  and  evaluation  is  a  valuable  part 
of  military  acquisition.  Mostly  because  of  the 
nature  of  flight  testing,  OT&E  of  aircraft  has 
several  important  limitations,  but  modeling  and 
simulation  holds  promise  to  overcome  these 
limitations  and  improve  the  quality  of  the 
operational  evaluation.  However,  the  needs  of  the 
test  and  evaluation  user  of  modeling  and 
simulation  may  be  different  from  the  more 
familiar  needs  of  the  other  communities,  such  as 
training.  Along  with  these  needs  comes  a  set  of 
challenges  that  the  M&S  community  must 
understand  and  overcome. 


8  One  such  report  is  the  "Study  on  the  Effectiveness  of 
Modeling  and  Simulation  in  the  Weapon  System 
Acquisition  Process,"  written  for  Deputy  Ehrector, 
Test  Facilities  and  Resources  by  A.  Patenaude,  D. 
Helmuth,  P.  Potter,  and  M.  Borowski,  October  1996 
(<http://www.acq.osd.mi1/te/pubdocs/acqstudy.h 
tm>). 
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Abstract 

Test  and  Evaluation  is  no  longer  a  stand  alone  activity  but 
is  considered  to  be  a  fundamental  and  integral  part  of  the 
Systems  Engineering  process  i.e.  the  overall  system 
design  and  manufacture.  By  considering  how  change 
comes  about  and  listing  aspects  of  change  in  Systems 
Engineering  at  large,  future  Test  and  Evaluation  practice 
has  been  investigated.  It  is  shown  that  there  are  many 
factors  to  explore  and  that  they  are  highly  interactive.  It  is 
clear  that  there  will  be  major  changes  and  that  no  simple 
action  can  be  taken  to  keep  abreast  of  change.  Improved 
training  and  qualification  for  T&E  practitioners  offers  the 
best  protection  against  an  uncertain  future. 

Specific  findings  are:- 

•  T&E  practitioners  will  be  integrated  into  Systems 
Engineering  teams. 

•  Education  and  Training  for  future  T&E  need  to  be 
reviewed  and  revised. 

•  Virtual  systems  will  become  the  normal  test 
environment. 

•  The  major  T&E  challenge  will  be  to  validate  the 
virtual  environments. 

•  Test  ranges  and  test  equipment  will  still  be  needed  but 
will  be  used  sparingly. 

•  Test  and  Evaluation  will  become  accepted  as  the  key 
validation  hub  providing  feedback  at  all  stages  of  a 
project. 

•  Test  and  Evaluation  will  be  in  use  to  test  all  manner  of 
variables,  including  the  subjective  soft  systems 
parameters. 

f  Test  engineers  will  need  to  be  familiar  with  the 
phenomenological  approach  of  Soft  Systems  as  well 
as  with  the  reductionist  approach  of  the  scientific  and 
engineering  disciplines. 

Overall,  enterprises  need  to  examine  their  attitude  to  T&E 
and  put  in  place  means  to  keep  up  with  the  Best  Practice 
and  be  able  to  contribute  toward  Future  Best  Practice. 


What  will  Systems  Engineering  practice  entail  in  the 
early  2U*  Century? 

The  Cost.  Time,  Performance  Factors 
Test  and  Evaluation  (T&E)  is  no  longer  ‘test  as  you  go’. 
It  is  fast  becoming  a  key  part  of  the  overall  process  Of 
how  systems  evolve  (Systems  Engineering).  To  see  what 
T&E  changes  can  be  expected  to  arise  it  is  first  necessary 
to  consider  the  likely  changes  in  Systems  Engineering. 

The  main  interacting  drivers  of  Systems  Engineering,  due 
to  market  pressiues,  are:  - 

•  Cost 

•  Time  to  market 

•  Performance  /  Contractual  compliance. 

These  Cost,  Time,  Performance  (CTP)  goals  have  always 
been  important  but  today  they  are  crucial  to  economic 
survival.  Improvements  have  been  achieved,  not  so  much 
by  tuning  at  the  edges,  but  by  applying  iimovative  and 
unexpected  thinking  on  how  to  optimise  all  of  these 
factors  at  once. 

Some  Ideals  of  System  Engineering  Methodology 

The  perfect  design  situation  would  allow  the  requirement 
to  be  met  with  zero  design  and  implementation  risks.  To 
do  this  it  strives,  but  never  reaches,  the  goals  of  having 
perfect:  - 

•  Assumptions 

•  Requirement  extraction 

•  Specification  development 

•  Design  process 

•  Test  and  evaluation 

•  Technology 

•  Human  interactions. 

Factors  of  change 

Processes  by  which  the  need  is  ipet  can  be  characterised 
as:  - 

•  Extrapolation  of  current  ways,  including  cross 
fertilisation 

•  Implementation  of  plausible,  science  fiction-like,  ideas 

•  Unexpected  mutations  of  the  process,  or  technology 
employed. 

Changes  in  human  attitude  due  to  cultural  workplace 
changes,  politics,  industrial  trends,  social  forces  or  war 
are  also  vital  factors. 

Extrapolation  of  current  wavs 

Examples  of  these  are  easily  seen  once  sufficient 
awareness  of  the  situation  is  obtained.  A  problem  for  the 
observer  is  that  some  major  changes  are  assuredly  being 
carried  out  in  relative  secrecy  inside  Corporations  and 
defence  circles.  One  cannot  be  fully  certain  of  where  the 
indushy  is  on  the  curve  of  improvement.  Once  the 
change  factor  had  a  time  constant  of  some  10-15  years  for 
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major  systems.  Recent  developments  have  shortened  this 

to  3-4  years  and  in  some  cases  to  1  year  or  less.  Even  less 

is  the  target  today. 

Obvious  factors  causing  change  are:  - 

•  Improved  Systems  Engineering  processes  involving 
increasingly  more  digitisation 

•  Progress  against  the  requirements  for  “Cost, 

'  Time  and  Performance”. 

•  Ownership  of  the  process:  suppliers  will 
control  the  after  sales  situation  as  their 
investment  in  models  and  staff  formation  is 
not  passed  on  to  the  buyer. 

•  Computing  techniques  will  improve 
interactions  but  will  complicate  the  T&E 
validation  process. . 

•  Integrated  Processes  and  Recurrent  Engineering 

•  Experiences  have  now  proven  the  worth  of 
integrated  processes  in  improving  all  of  the 
CTP  factors.  Enterprises,  at  all  levels  of 
scale,  are  adopting  this  more  common-sense 
approach. 

•  Virtual  operation  of  development 

•  Already  in  use,  but  being  slowed  by  lack  of 
proven  techniques  for  running  computer 
systems  in  a  virtual  network  at  the  designer’s 
knowledge  level. 

•  Global  distances  not  a  factor.  Examples  are 
India  used  to  service  software  and  printing; 
Motorola  making  use  of  Internet  to  control 
CTP  factors  of  work  taking  place  across  the 
globe. 

•  Travel  to  be  physically  close  will  lessen  as 
virtual  reality  allows  equivalent  experiences 

•  Virtual  experience  becomes  the  norm. 

•  Reuse  allows  evolution  based  on  previous  generation 

•  Well  understood  in  Japanese  consumer 
electronics  and  car  industry.  Now  being 
adopted  in  defence  sector  via  COTS  and 
evolving  designs. 

•  Systems  of  Systems  -  emergent  properties  the  issue 

•  Many  systems  are  now  mature  with  little  real 
change  in  how  they  are  used  and  interact  with 
other  systems  -  cars,  teleconummication  add¬ 
ons,  white  goods,  computing  such  as  personal 
computers. 

•  Front  End  Loading  (FEL)  becoming  the  focal  area  for 
effort 

•  CTP  forces  demand  less  risk  in  design 

•  Much  more  efficient  to  spend  more  effort  at 
FEL  stage  than  correct  errors  later 

•  Engineers  will  need  to  be  more  familiar  with 
FEL  work  and  problem  solving;  they  will  be 
less  involved  in  the  detailed  design  that  they 
are  taught  today 

•  There  is  an  Academic  Engineering  support 
lag  of  around  10  years  here.  Research 
programmes  and  syllabuses  are  trapped  in 
outmoded  paradigms. 

•  Minimal  time  to  market  concepts  are  in  vogue 

•  Boeing’s  6  month  plane  concept 


•  2  day  house 

•  Demand  for  this  mode  of  thinking  is  high  as 
reduced  time  to  market  cuts  costs  of  design, 
puts  the  system  into  service  faster  and  avoids 
customer  requirement  changes  and  being 
second  in  the  market  place.  Safety  and  other 
performance  issues  are  in  danger  of  neglect. 
This  places  far  higher  demands  on  error  fi'ee 
FEL  work. 

•  Greater  automation  of  latter  stage  design 

•  Much  of  today’s  sub-systems  developments 
are  done  using  design  tools  that  are 
increasingly  becoming  automated  and  that 
interface  digitally  to  the  digital  factory. 

•  Reduced  manufacture  of  prototypes 

•  Designs  held  in  digital  form  until  the  final 
physical  system  is  needed. 

•  Less  need  to  build  test  objects  to  see  if  they 
are  acceptable  and  ready  for  a  new  market  or 
military  requirement. 

•  Smarter,  more  aggressive,  customers 

•  Highest  ever  expectations 

•  Competition  and  ‘bottom  line’  costing 
attitudes  give  the  customer  a  better  edge  over 
the  client. 

•  Ever  increasing  computer  power 

•  Knowledge  processing  now  faster  and  better, 
no  limit  to  be  seen  ahead 

•  Greater  detail  in  simulations 

•  Easier  to  use  and  train  to  use. 

•  Adoption  of  knowledge  based  thinking  in  systems 
design  -  as  opposed  to  data  processing  thinking 

•  Knowledge  based  systems  were  slow  to 
emerge  as  they  confront  the  reductionist 
thinking  of  engineers  and  hard  scientists. 

•  Increase  in  legal  requirements 

•  Hundreds  of  Environmental,  Occupational 
Health  and  Safety,  Trade  and  Industry  and 
other  laws  increasingly  impact  on  systems 
development  and  operation.  Physical  testing 
getting  harder  and  more  expensive  to  carry 
out. 

•  The,  adversarial,  litigious  society  demands  higher 
degree  of  expectation  of  supply 

•  Law  suits  have  become  the  answer  for 
disappointment 

•  Legal  liability  is  a  minefield.  T&E  process 
can  assist  in  limiting  claims 

•  Settlements  can  bankrupt  a  Systems  House 

•  Increasingly  harder  to  predict  the  outcome  of 
legal  suits. 

•  Change  and  re-engineering  of  work  forces 

•  Accepted  as  normal 

•  Constant  state  of  apparent  improvement 
currently  ignores  human  factor. 

•  Union  forces  not  holding  length  of  tenure 

•  Vendor  able  to  be  take  more  risks  as 
workforce  costs  are  more  flexible 

•  Short  contracts  give  workers  less  security 
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•  Vendor  system’s  development  and  continuity 
history  is  weakened,  especially  in  software 
areas. 

•  Simulated  Test  and  Evaluation  Process  of  US  DoD 

•  Processes  in  defence  must  be  open  thus  being 
at  odds  with  CPT  factors 

•  STEP  is  latest  showing  of  major  shifts  in 
process. 

It  is  to  be  noted,  however,  that  many  of  the  issues  listed 
above  cannot  be  met  by  the  discipline  of  engineering  as  it 
is  constituted  today.  Many  other  disciplines  are  involved. 
Systems  Engineers  need  to  be  broader  in  horizon. 
Multidisciplinary  teams  (that  is,  those  containing 
contributions  fi'om  the  Arts,  Sciences  and  Engineering  - 
not  just  a  mix  of  different  branches  of  engineering)  are 
essential.  They  require  visionary  leaders  who  are 
supported  by  people  able  to  synthesise  and  analyse  the 
innovative  measures  suggested  by  the  leaders. 

Systems  Engineering  will  continue  to  evolve  and  keep 
seeking  improvement  as  the  many  forces  at  play  flex  with 
time. 

Imnlementation  of  plausible  science  fiction-like  ideas 

These  cannot  be  forecast  with  anywhere  the  level  of 
certainty  of  the  extrapolated  parameters.  Although  these 
are  easy  to  suggest  it  is  has  been,  to  date,  only  the  science 
fiction  authors  and  cartoonists  that  have  been  able  to 
mould  these  into  virtual  scenarios.  Virtual  reality 
computer  based  methods  now  allow  such  fantasies  to  be 
exercised  to  see  how  they  might  work  in  dynamic 
situations. 

A  few  of  the  obvious  developments  that  can  be 
foreshadowed  as  plausible,  but  for  which  it  is  veiy 
imcertain  as  to  when  they  will  come  into  practice,  are  now 
listed.  They  may  appear  to  be  somewhat  far  off  of  the 
topic  of  21'*  Century  Systems  Engineering  practice  but 
their  description  paints  the  kind  of  scenarios  that  could 
arise  illustrating  that  the  situation  could  be  quite  different 
to  the  extrapolated  future  described  above.  Ihe  future  will 
certainly  reveal  some  most  unexpected  changes.  An 
eminent  physicist  in  1950  suggested  that  the  transistor 
might  find  limited  use  to  make  smaller  walkie-talkie 
radios  but  apart  fi-om  that  it  was  hard  to  see  where  it 
would  have  much  impact! 

Examples  of  the  plausible  great  changes  are:- 

•  Chemical  signal  processing  replaces  the  current  non¬ 
living  physical  method  massively  reshaping  the  IC, 
electronic  systems  and  computing  industry.  Features 
of  these  information  processing  systems  would  be:- 

•  Self  repairing 

•  Self  evolving 

•  Self  reproducing 

•  Require  radically  different  means  of  design, 
use  and  manufacture 

•  CTP  drivers  again  improved 


•  Gene  engineering 

•  Improved  performance  of  living  things 

•  Different  human  class 

•  Potential  deleterious  effect  on  humans 

•  Anti-gravity  suspension  using  low  energy 

•  Drastic  changes  to  transport  systems  and 
equipment 

•  Practical  use  of  yet  undetected  minor  physical  forces 

•  Gravity  waves  give  new  communications 
medium 

•  ESP  and  the  like  with  humans  able  to  read 
minds 

•  Human  thinking 

•  Changed  directions  for  scarce  resources  uses 

•  Major  wars  with  physical  or  trade  weapons 

•  A  less  selfish,  more  caring,  society  might 
emerge. 

Unexpected  mutations  of  the  process  or  technology. 

True  mutations  of  note  cannot  really  be  forecast  leaving 
us  with  unscheduled  changes  that  can  have  potentially 
great  impact  -  for  both  the  good  and  the  bad.  These  can  be 
totally  impredictable  despite  the  best  use  of  expert’s  ideas 
through  Delphi  forecasting  techniques.  Mutations  have  to 
be  allowed  for  as  they  occur.  Fortunately,  they  usually 
give  us  some  notice  of  the  change  and  gradually  move 
from  the  fantastic  class  into  the  extrapolated  kind  of 
change. 

Current  State  of  T&E  Practice 

Test  and  Evaluation  is  carried  out  somewhere  between 
two  extremes  of  philosophy. 

In  the  ‘experiential’  approach  a  ‘build  and  test’  sequence 
is  followed  to  see  if  the  sub-systems  built  do  perform  to 
test  requirements  that  tend  to  be  decided  upon  by 
historical  practice  and  local  appreciation  of  the  system 
need.  Here  the  practitioner’s  confidence  stems  from  their 
professional  experience  but  there  has  been  no  way  of 
satisfactorily  proving  that  it  is  right.  This  approach 
sufficed  when  the  Idad  times  were  long  and  the  TCP 
parameters  were  not  under  the  extreme  pressure 
experienced  today.  Shorter  time-scales,  better 
documentation  and  widely  distributing  the  process  detail 
makes  it  harder  to  hide  errors  or  lay  blame  elsewhere. 

In  the  ‘Master  Planned’  approach,  T&E  is  made  a  focal 
part  of  every  stage  of  the  whole  process  from  conception 
to  taking  the  system  out  of  service.  Tests  collect  key 
information  that  enables  evaluation  of  performance 
according  to  an  overall  plan  set  up  from  the 
commencement  of  the  project.  The  tests,  and  the  use  of 
test  data  in  evaluations,  are  set  up  to  map  upward  to  the 
give  indicators  of  success  at  meeting  the  requirement  at 
the  operational  level. 

Another  limiting  factor  about  current  T&E  is  that  there  is 
still  little  requirement  for  formally  certificated  T&E 
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qualifications  in  which  practitioners  are  assessed  as  to 
their  experience,  educational  and  skill  levels. 

T&E  make  use  of  veiy  expensive  test  facilities  in  which 
many  tests  are  carried  out.  Tests  are  often  used  to  get 
stages  signed  off  rather  than  using  to  inject  wisdom  into 
the  FEL  stages.  As  recognised  above,  this  is  will 
increasingly  change. 

T&E  in  the  Early  21*‘  Century 

Obviously  T&E  is  a  major  discipline  with  increasing 
importance  in  controlling  CTP  risk  factors.  As  Systems 
Engineering  Best  Practice  changes  then  so  will  the  need 
for  T&E  practice  to  change  and  to  contribute  to  key 
project  decisions.  It  can  be  argued  that  T&E  will  be  a 
more  important  activity  than  it  has  been  seen  to  be  to  date. 
It  seems  to  be  a  truism  that  the  T&E  personnel  are  able  to 
assist  ensure  the  system  design  is  ‘right’  more  than  any 
other  discipline  involved. 

It  is  not  possible  here  to  comment  on  all  issues  raised 
earlier  in  this  paper.  Scanning  the  list  of  changes  in 
Systems  Engineering  methodology  shows  that  certain 
issues  are  dominant. 

Overall  one  thing  becomes  clear  -  the  various  change 
parameters  are  inter-related  to  a  large  extent  showing  that 
the  best  practice  will  be  constantly  changing  with  time 
constants  of  utility  reducing  to  a  matter  of  a  year,  or  less, 
for  a  given  set  of  practices.  Practices  could  therefore 
need  to  change  within  the  life  of  project  or  be  capable  of 
be  completed  ever  more  quickly! 

A  few  factors  of  change  are  now  considered  giving  brief 
outlines  of  how  T&E  will  change  because  of  them. 

Digitisation  and  Virtual  operation 

Placing  the  systems  design  into  a  computer-based 
simulation  has  required  dramatic  change  in  the  attitude  to 
design  and  evaluation.  A  decade  ago  the  then  available 
level  of  Modelling  &  Simulation  (M&S)  was  not  really 
suited  to  the  high  level  system  decisions  that  had  to  be 
taken.  M&S  now  supports  scenario  building,  prototyping, 
subsystems  design,  manufacturing  try-outs,  machine  tool 
and  production  line  set  up  and  control  of  manufacturing. 
It  also  supports  servicing,  configuration  management, 
training  and  more.  It  will  be  able  to  do  even  more  in  the 
future  and  will  form  a  key  element  of  any  contracted 
procurement. 

The  demands  for  test  ranges  and  equipment  has  changed 
significantly  due  to  adoption  of  digitisation.  The  changes 
emerging  are:- 

•  Full  scale,  long  period  physical  testing  will  be  less 
practical 

•  Live  fire  and  other  ‘final’  testing  will  become 
environmentally  imacceptable 


•  More  special,  rapidly  configured,  test  set  ups  are 
needed 

•  Test  personnel  will  need  enhanced  problem  solving 
and  digital  skills  and  will  be  an  integral  part  of  the 
Systems  Engineering  team. 

•  Test  personnel  need  improved  training  and  recognition 
if  they  are  to  be  accepted  within  the  teams 

•  Computer  environments  will  require  validation  at  all 
stages  of  development  requiring  new  forms  of  test  bed 
used  in  the  distributed  interactive  mode 

•  Nodes  of  the  system  design  process  must 
communicate  unambiguous  design  knowledge,  not  just 
data  sets. 

Integrated  Programmes  and  Concurrent  Engineering 
Methods 

Bringing  more  functions  together  earlier  has  the  potential 
to  uncover  design  flaws  where  they  are  far  less  costly  to 
fix.  This  paradigm  has  become  known  as  the  Integrated 
Product  and  Process  Development  (IPPD)  method.  When 
a  representative  group  of  experts  are  assembled  as  a  team 
they  are  able  to  develop  better  appreciation  of  the 
interactions  within  the  design  situation.  They  must  be 
careful  to  ensure  that  their  decisions  are  sound  and  that 
risks  are  understood  and  minimised.  The  concept  has  been 
proven.  The  pivotal  role  of  T&E  is  clearer  m  this 
methodology;  it  is  used  as  a  source  of  wisdom,  not  just  as 
an  ‘after  the  fact’  test  to  prove  things  are  in  order. 

Some  eight  years  after  the  integrated  development 
approach  was  first  implemented,  the  US  Department  of 
Defence  has  gathered  the  best  of  the  IPPD  process  into 
the  Simulated  Test  and  Evaluation  Process  (STEP).  This 
has  compacted  the  expected  milestones  time  span  and, 
most  importantly,  placed  evaluation  at  the  hub  of  each 
stage  of  the  process.  They  have  made  it  clear  that  T&E  is 
a  central  feature  of  all  stages. 

This  methodology  brings  with  it  greater  need  for  means  to 
better  evaluate  subjective  decisions,  soft  systems, 
requirements  and  specifications  as  well  as  the 
performance  of  conventionally  engineered  artefacts.  It 
also  calls  for  ways  to  evaluate  models  at  their  various 
stages  of  development. 

As  much  of  the  system  is  in  a  digital  simulation  form  the 
testing  is  now  different.  Physical  test  equipment  will  still 
be  needed  but  now  they  are  not  facilities  that  are  used  for 
prolonged  periods  but  need  to  be  able  to  be  set  up  rapidly, 
under  computer  control,  to  carry  out  specific  routines 
needed  to  verify  point  profiles  in  the  performance  of  the 
digital  model. 

At  present  there  is  low  confidence  in  the  model  as  the 
final  representation.  It  is  clear,  however,  that  models  are 
fast  becoming  reliable  equivalents  of  the  real  thing  -  as 
long  as  they  are  calibrated  appropriately  and  used 
intelligently. 
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Education  and  Training 


Assuming  that  most  of  the  development  process  will 
reside  in  the  digital  domain  then  T&E  personnel  will  need 
to  train  appropriately.  Looking  forward  a  decade  we  can 
expect  they  will  need  to  be  capable  in  virtual  modelling, 
in  the  testing  of  models  and  when  and  how  to  use  physical 
test  facilities  to  improve  the  model. 

It  is  clear  that  education  and  training  is  a  vital 
tmderpinning  activity  for  all  of  the  change  parameters. 

Facilities 

Legal  and  economic  forces  vvill  work  to  make  it  harder  to 
keep  large  facilities  in  a  state  of  the  readiness  needed. 
There  will  be  need  to  condense  range  assets  and 
investment  which  will  be  in  conflict  with  the  need  to  have 
access  to  test  facilities  with  adequate  accuracy  to  ensure 
the  points  in  the  model  have  low  enough  uncertainty,  and 
good  automation  and  digital  interfaces.  Study  is  needed  to 
see  how  the  many  conflicting  factors  can  be  harmonised 
to  ensure  the  models  can  be  kept  well  calibrated. 

Systems  of  Systems 

It  is  now  clear  that  the  larger  systems  should  be  viewed  as 
Systems  of  Systems  (SoS)  in  which  each  system  interacts 
via  its  own  emergent  properties.  Thus  System  design  has 
to  cope  with  more  system  level  blocks  but  with  reducing 
interest  in  their  internal  behaviour. 

This  change  in  perspective  to  ‘whole  of  enterprise’ 
modelling  requires  that  the  individual  models  interface 
efficiently.  Testing  interfaces  and  SoS  models  requires  a 
quite  different  kind  of  capability  to  that  of  the  software 
and  equipment  tester  of  today. 

Minimal  Time  Concepts  and  Emphasis  nn  FET , 

Increasing  the  effort  in  the  FEL  area  needs  high  budget 
loading  at  the  early  stages.  Governments  seem  to  find  this 
hard  to  set  up  for  it  runs  counter  to  many  of  their 
budgeting  processes.  Indeed  maintaining  team  knowledge 
when  projects  are  brought  to  fruition  sooner  and 
replacements  sought  less  frequently,  will  require 
improved  knowledge  capture  and  transfer.  The 
commercial  world  already  faces  this  dilemma  but  has  to 
do  it  to  compete.  Government  procurement  must  change 
to  provide  high  FEL  as  diis  is  a  major  key  to  meeting  CPL 
factors. 

Again  we  see  that  Test  and  Evaluation  personnel  need  to 
be  trained  to  be  able  to  contribute  well  in  FEL  work.  This 
requires  quite  different  training  to  that  given  today. 

Knowledge  Capture 

Reduced  programme  time-scales,  fewer  less  frequent 
projects,  increased  use  of  existing  items  and  data,  all  point 
to  the  need  to  develop  enhanced  knowledge  capture  and 


transfer.  For  T&E  this  will  increasingly  mean  that  revised 
analysis  of  previous  testing  will  become  politically 
necessary  and  more  cost  effective  than  new  tests. 
Environmental  issues  will  cause  all  new  real  testing  to 
have  to  be  justified.  Increasingly  sub-contractors  that  can 
supply  “certified”  products  in  short  time  scales  will  have  a 
major  competitive  edge.  This  will  put  further 
requirements  on  parts  of  the  T&E  community  who  will 
need  to  imderstand  the  reworked  testing  and/or  keep 
records  of  their  own. 

Himians  in  the  process 

The  final  aspect  to  be  covered  here  is  that  the  thinking 
processes  of  the  human  mind  are  becoming  identified 
more  with  the  new  processes.  It  is  becoming  clear  that  the 
ability  to  synthesise  new  ways  in  a  supportive 
environment  is  needed.  Humans  in  Systems  Engineering 
cannot  be  treated  as  deterministic  processors.  Placing 
them  into  rigid  processes  where  Capability  Maturity 
Models  and  Quality  processes  are  expected  to  yield 
optimum  overall  team  performance  has  now  been  seen  to 
have  its  shortcomings.  Soft  Systems  thinking  is  needed 
and  this  uses  different  means  of  inquiry  to  engineering 
and  science.  Test  and  Evaluation  personnel  are  rarely 
equipped  to  evaluate  this  aspect  of  systems. 

Conclusion 

Test  and  Evaluation  is  no  longer  a  stand  alone  process  but 
is  considered  to  be  a  fundamental  and  integral  part  of  the 
overall  system  design  and  manufacture  Systems 
Engineering  activity.  By  considering  how  change  comes 
about  and  listing  aspects  of  change  in  Systems 
Engineering  at  large,  future  Test  and  Evaluation  practice 
has  been  investigated.  It  is  shown  that  there  are  many 
factors  to  explore  and  that  they  are  highly  interactive.  It  is 
clear  that  there  will  be  major  changes  and  that  no  simple 
action  can  be  taken  to  keep  abreast  of  change.  Improved 
training  and  qualification  for  T&E  practitioners  offers  the 
best  protection  against  an  imcertain  future. 

Specific  findings  are:  - 

•  T&E  practitioners  will  be  integrated  into  Systems 
Engineering  teams. 

•  Education  and  Training  for  future  T&E  need  to  be 
reviewed  and  revised. 

•  Virtual  systems  will  become  the  normal  test 
environment 

•  The  major  T&E  challenge  will  be  to  validate  the 
virtual  environments. 

•  Test  ranges  and  test  equipment  will  still  be  needed  but 
will  be  used  sparingly. 

•  Test  and  Evaluation  will  become  accepted  as  the  key 
validation  hub  providing  feedback  at  all  stages  of  a 
project. 

•  Test  and  Evaluation  will  be  in  use  to  test  all  manner  of 
variables,  including  the  subjective  soft  systems 
parameters. 
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•  Test  engineers  will  need  to  be  familiar  with  the 
phenomenological  approach  of  Soft  Systems  as  well 
as  the  reductionist  approach  of  the  scientific  and 
engineering  disciplines. 

Overall,  enterprises  need  to  examine  their  attitude  to  T&E 
and  put  in  place  means  to  keep  up  with  the  Best  Practice 
and  be  able  to  contribute  toward  Future  Best  Practice. 
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Abstract 

A  desire  for  future  instrumentation  is  the  ability  to 
provide  data  for  both  test  and  training  users  during  a 
single  event.  Conventional  wisdom  assumes  that 
there  are  significant  advantages  to  performing  both 
together.  Efficiencies  of  resource  use  (time,  people, 
money  and  equipment)  are  the  expected  advantages. 
The  design  of  a  single  tool  (read:  instrumentation 
system)  seems  feasible.  This  paper  proposes  a 
conceptual  tool  that  is  basic  enough  to  be  valuable  to 
both  communities  without  being  expensive,  complex, 
biased  or  restrictive. 

A  need  common  to  both  users  is  position  of 
participants.  The  instrumentation  design  is  then 
determined  by  answers  to  the  following  questions. 
How  many  participants  need  to  be  monitored?  What 
are  the  types  of  participants?  Where  is  the  event 
performed  (size,  location  ^d  environment  of  the 
sandbox)?  What  accuracy  is  needed?  When  and 
where  is  the  data  needed  (feedback,  display, 
archiving)?  What  will  the  tool  cost?  When  can  I  use 
it?  What  are  its  care  and  feeding  requirements? 

Past  and  existing  systems  had  specific  needs  based 
on  performing  a  specific  task.  Testers  and  trainers 
are  different  fi'om  each  other.  Their  needs,  events  and 
methods  are  different.  I  am  proposing  a  time,  space, 
position  information  (TSPI)  system  concept  that  will 
be  sufficient  for  both  groups  in  the  majority  of  their 
endeavors. 

The  proposed  system  will  use  two  technologies  to 
achieve  its  objective:  providing  suitable  and  efficient 
TSPI  for  both  types  of  users  during  a  single  event. 
The  two  technologies  are  the  Global  Positioning 
System  (GPS)  and  a  low-earth-orbit  satellite  system. 
The  first  is  capable  of  determining  suitable  and 
efficient  TSPI.  The  second  is  potentially  capable  of 
communicating  that  data,  suitably  and  efficiently,  to 
its  appropriate  destination. 


Introduction 

This  document  describes  an  instrumentation  concept, 
suitable  for  both  test  and  training  events. 

That  simple  sentence  conjures  many  different  and 
unique  visions.  A  clarification  of  the  terms  is  thus 
necessary  as  a  precursor  to  the  description  of  the 
instrumentation  concept.  This  concept  is,  as  Webster 
defines  it,  a  general  idea  or  understanding  derived 
from  specific  events. 

Test 


A  test  event  seeks  knowledge.  Cause  and  effect  are 
determined  by  adjusting  variables,  obtaining 
appropriate  data  with  a  set  of  instrumentation  and 
analyzing  that  data.  The  test  methodology  must  be 
appropriate  for  the  objective  sought.  Tests  normally 
occur  in  a  sandbox.  My  vision  of  a  sandbox  is  an 
environment  conducive  to  achieving  the  objective 
information.  It  may  be  an  open-air  range  having  all 
necessary  manpower,  equipment,  instrumentation, 
data  reduction,  needed  feedback  loops  and  displays. 
It  may  also  be  a  single  tool  that  stimulates  an 
electronic  circuit  and  records  the  response.  A 
computer,  running  models  having  well  understood, 
well  documented  and  valid  algorithms  can  also  be  a 
sandbox.  There  are  many  more  sandboxes.  The 
objective  of  a  test  is  confirmation  of  hypotheses. 
How  does  something  work?  How  well  does  it  work? 
What  are  the  conditions  when  it  works  less  well  or 
not  at  all?  What  can  degrade  performance  or  stop  it 
from  working?  What  care  and  feeding  procedures  are 
appropriate  and  how  do  they  effect  performance? 

Training 

A  training  event  teaches  lessons  that  are  based  on  the 
knowledge  gained  through  testing.  It  spans  the 
instructional  gamut  from  individual  learning  to  large- 
group  learning,  and  includes  many  desired  outcomes. 
It  also  occurs  in  sandboxes,  although  the 
characteristics  of  a  training  sandbox  may  differ  jfrom 
characteristics  of  a  test  sandbox.  Feedback  is  often 
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vital,  suggesting  realistic  latency  needs.  The 
information  from  a  training  event  (obtained  from 
instrumentation)  must  reinforce  good  behavior  and 
piuiish  bad  behavior. 

These  are  simple  descriptions.  Both  types  of  events 
need  to  be  well  understood  and  well  dociunented  if 
they  hope  to  achieve  their  respective  objectives.  They 
differ  in  scope,  resources,  management,  control, 
structure  and  responsibility. 

Instrumentation  Systems 

Instrumentation  is  the  defining  factor  of  test  because 
it  alone  produces  information.  Objectives  are 
obtained  only  by  analyzing  that  information. 

Training  is  possible  without  instrumentation,  but  the 
absence  of  data  makes  achievement  and  measurement 
difficult. 

The  first  question  in  either  endeavor  should  be  “What 
do  I  want  to  learn  or  teach?”  The  answer  to  that 
question  defines  the  quality  and  quantity  of 
information  that  is  necessaiy.  Instrumentation 
determines  the  data  extracted  from  the  test  or  training 
implementation. 

Has  instrumentation  changed  over  the  past  25  years 
of  my  experience  in  testing  and  a  lifetime  of 
learning?  My  answer  is  an  admittedly  ambiguous 
“yes  and  no”.  Changes  are  continuously  being 
proposed  and  implemented  in  the  quest  for  improved 
performance.  Technological  advance  may  improve 
our  ability  to  obtain  information  that  was  previously 
unobtainable.  The  constants  are  less  noteworthy  but 
no  less  important.  We  still  use  many  procedures  and 
systems  that  have  existed  for  decades  or  generations. 
Many  concepts  seem  ageless.  “Faster,  better  and 
cheaper”  are  the  rewards  for  appropriate  change  but 
the  principles  remain  fixed. 

Combining  teat  and  training 

The  current  test  and  training  quest  seems  to  be  a 
desire  to  combine  the  two  events.  I  was  asked  to 
prepare  this  paper  based  on  the  stipulation  that 
obtaining  test  data  in  a  training  event,  or  training 
data  in  a  test  event,  was  desirable.  My  experience 
does  not  allow  that  stipulation.  I  have  aecomplished 
and  witnessed  tests  with  dual  users.  Some  work  well 
and  others  don’t.  Training  of  some  type  results 
whenever  a  task  is  performed,  but  the  lessons  learned 


may  be  good,  bad  or  indifferent.  I  have  never  seen  a 
precise  determination  of  cause  and  effect  during  a 
training  event  because  control  of  variables  is 
normally  insufficient.  A  case  by  case  analysis  of  the 
goals  and  procedures  of  any  one  event  may  yield 
speeific  instances  where  data  are  appropriate  for 
both  testers  and  trainers,  but  universal  benefits  to 
both  groups  from  a  single  event  have  not  been 
documented.  Bias  is  very  likely  to  exist  in  data  that 
results  from  any  event.  Understanding  the  nature  of 
that  bias  (time,  participant  qualities,  environment, 
methodology,  event  control,  procedures,  and 
incentives,  to  name  a  few)  is  important.  Bias  can  be 
overcome  only  if  the  event  is  well  imderstood,  well 
managed  and  well  documented,  which  may  be 
costlier  or  less  effective  than  two  separate  events. 

Combination  of  test  and  training  is  not  a  panacea  to 
reduced  cost/time/people/equipment  (read: 
resources).  The  desire  to  accomplish  more  with  less 
is  commendable  and  the  propensity  to  state  “here  is 
the  answer”  is  rmderstandable.  Limitations,  risks  and 
consequences  of  combination  must  be  thoroughly 
considered  and  documented.  I  would  be  glad  to 
receive  and  debate  the  thoughts  of  any  reader  on  the 
pros  and  cons  of  such  a  combination. 
(burski@mindspring.com)  I  make  that  offer 
confident  that  the  concerned  audience  for  this  subject 
is  small. 

Is  there  an  instrumentation  concept  that  might  allow 
generation  of  suitable  data  for  different  users  during 
a  single  event?  I  believe  the  answer  to  that  question  is 
“yes”  with  the  following  limitations. 

Concept  Scope 

I  will  limit  the  scope  of  this  report  to  test  or  training 
events  that  occur  with  actual  participants.  These 
events  might  take  place  on  any  suitable  real  estate, 
worldwide,  including  ships  and  simulators.  The 
participants  may  be  a  few  soldiers  or  vehicles  from  a 
single  military  Service,  or  they  may  be  a  thousand- 
plus  entities  opposed  in  joint  exercises. 

The  proposed  instrumentation  system  will  be 
described  in  general  terms  only.  My  experience  in 
responding  to  request-for-proposals  is  that  strict, 
detailed,  direetive  criteria  are  not  conducive  to 
success  because  innovation  by  experts  and  “out-of- 
the-box”  novices  is  precluded.  People  are  ingenious 
and  inventive  when  they  are  allowed  latitude  in 
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problem  solving.  Guidelines  are  appropriate  to  avoid 
historical  problems  but  they  can  quickly  become 
inflexible  and  stifling. 

Concept  Requirement 

Data  Suitable  for  Two  Users 

What  data  is  necessary  for  both  testers  and  trainers? 
Time  and  position  of  participants  meet  that  criteria. 

It  is  not  a  new  or  original  idea  to  restrict  an 
instrumentation  system  to  time,  space,  position 
information  (TSPI)  but  my  primary  objectives  in 
providing  data  to  two  users  are  simplicity,  validity 
and  universality. 

Why  is  all  data  but  TSPI  precluded?  An  example  of 
information  normally  desired,  but  not  included  in 
this  instrumentation  concept,  is  scoring  of  weapons. 
Scoring  is  now  accomplished  in  many  different  ways 
by  many  different  instrumentation  systems.  Scoring 
requires  a  considerable  amount  of  weapon  setting, 
aiming,  trajectory,  coimtermeasure  and  firing 
information,  violating  my  goal  for  simplicity.  Pairing 
is  reasonably  valid  scoring  for  only  a  few 
weapon/target  combinations.  Scoring  is  available 
today  (with  a  few  exceptions)  only  on  instrumented 
ranges  and  not  in  diverse  locations,  including  urban 
environments,  that  exist  off-range. 

A  single  quality  and  quantity  of  score  information  is 
not  likely  to  satisfy  the  requirements  of  two  different 
users.  Scoring  that  is  sufficient  for  training  may  not 
accomplish  a  test  objective,  and  vice  versa.  A 
middle-ground  compromise  may  result  in 
conclusions  that  are  incorrect  or  misleading  for  both 
groups. 

I  oppose  adding  scoring  to  this  instrumentation 
concept  because  high  quality  scoring  data  is  difficult 
to  achieve.  It  is  expensive,  complicated,  restrictive  to 
normal  task  performance,  bandwidth  hungry, 
bimched  in  time,  feedback  inducing  and  quite  nasty 
to  validate.  If  scoring  data  is  not  of  the  highest 
quality,  it  is  incapable  of  supporting  several  users 
because  the  data  will  likely  be  insufficient  for  many 
needs.  That  does  not  seem  cost  effective.  Limitations 
of  the  instrumentation  will  be  quickly  forgotten  (if 
ever  learned)  and  it  will  be  used  for  inappropriate 
objectives,  i.e.,  it  will  provide  biased  or  incorrect 
data. 


The  same  rationale  is  true  for  all  information  except 
TSPI.  TSPI  is  the  only  data  that  may  satisfy  many 
users  from  a  single  event,  with  one  set  of 
characteristics. 

The  instrumentation  system  should  also  have  the 
following  additional  characteristics:  accuracy; 
reliability;  value  and  low-cost.  I  will  briefly  discuss 
my  understanding  of  each  of  these  characteristics. 

Instrumentation  Characteristics 

Simplicity  An  instrumentation  system  is  simple  if  it 
consists  largely  of  existing  technology.  I  don’t  want 
it  to  be  ten  years  in  development  and  difficult  to 
achieve.  I  also  don’t  want  it  to  be  cumbersome  to  a 
participant.  The  participant  must  perform  his  or  her 
tasks  in  a  manner  that  is  identical  to  performance 
without  the  instrumentation  system.  Simplicity  also 
implies  reliability,  which  means  ease  of  installation, 
initialization,  diagnosis  and  repair.  It  must  have  a 
high  utilization  rate.  Any  system  that  loses  half  its 
data  or  is  inoperative  half  the  time  is  quite  worthless 
because  the  important  data  may  be  that  which  is 
missing. 

Validity  The  data  must  be  based  on  truth.  For  this 
instrumentation  concept,  data  is  time  and  position. 
How  is  time  data  achieved?  How  is  position  data 
achieved?  What  are  all  sources  and  all  algorithms 
used  in  data  accumulation,  from  the  first  sensor 
reading,  through  transmission,  to  ultimate  display, 
integration  or  archiving?  What  manipulations, 
translations,  shortcuts  and  assumptions  are  made  in 
the  accumulation  of  infonnation? 

Universality  The  instrumentation  must  be  usable  by 
all  appropriate  participants,  in  any  location  on  the 
surface  of  the  earth.  An  aircraft  might  not  be  an 
appropriate  participant  because  aircraft  certification 
is  complex  and  aircraft  operating  conditions  are 
extreme.  Existing  aircraft  instrumentation  may  be 
able  to  provide  suitable  time  and  position  data  for 
integration  with  data  obtained  using  this  proposed 
concept.  “Any  location”  means  that  natural  or 
manmade  obstructions  and  interference  should  not 
preclude  the  instrumentation  system  from  achieving 
its  goal,  the  collection  of  information.  If  fact,  the 
latter  criteria  is  unlikely  to  be  absolutely  achieved, 
no  matter  what  the  cost.  A  reasonable  compromise 
between  performance  and  cost  may  be  that  some 
definable  obstruction  or  interference  is  acceptable. 
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Accuracy  The  instrumentation  system  must  produce 
data  accurate  to  some  definable  measurement,  for 
some  definable  percentage  of  its  operating  time,  with 
some  definable  level  of  confidence.  I  would  allow  the 
manufacturers  to  set  accuracy  parameters  they 
believe  are  achievable  and  then  enforce  strict 
warranty  procedures.  As  an  approximation,  I  would 
expect  X,  Y  and  Z  positions  to  be  within  three 
meters,  time  to  be  within  one  tenth  of  a  second, 
operating  percentages  in  the  mid  nineties  and  a 
confidence  of  95%. 

Reliability  This  characteristic  is  common  to  all 
military  equipment  used  in  the  field.  It  must  be 
robust.  It  should  work  when  it  is  turned  on  and 
perform  in  a  harsh  and  unpampered  environment.  It 
should  be  easy  to  determine  suitable  operation. 
Troubleshooting  and  repair  should  be  relatively 
quick  and  simple. 

Value  The  potential  benefit  to  be  gained  by  any 
system  is  much  more  difficult  to  define  and  measure 
than  the  cost,  but  no  less  important.  What  is  the 
value  of  any  piece  of  information?  What  is  the 
relative  value  of  one  piece  compared  to  another? 

Can  an  improvement  in  knowledge  or  an  increase  in 
readiness  be  quantified  and  priced  in  terms  of 
dollars,  people,  time  dr  equipment  (resources)?  As  a 
pilot  in  combat,  I  resented  being  the  measure-of- 
effectiveness  for  weapon,  tactics  or  countermeasure 
improvements.  I  was  pleased  that  improvements 
were  being  pursued  in-theater  but  I  was  appalled 
that  many  pursuits  did  not  occur  before  the  War.  On- 
the-job  training  is  a  costly  endeavor,  as  the 
proponents  of  TACTS/ACMIs  documented  in  their 
quest  for  improved  instrumentation  thirty  years  ago. 

Total  Cost  This  is  nothing  more  than  life-cycle  cost. 
My  concern  is  that  standard  DoD  descriptions  of 
system  total  costs  are  often  hazy  to  the  average 
citizen.  Obscuration,  for  many  different  reasons, 
often  seems  the  real  objective  of  defense  accounting 
procedures.  What  resources  are  really  expended  to 
obtain,  field,  maintain  and  operate  a  system? 

Latency  is  conspicuous  because  of  its  absence  as  a 
characteristic  of  this  instrumentation  concept.  As 
stated  above  in  the  Simplicity  section,  information 
from  this  instrumentation  system  should  not  be  used 
to  perform  operational  functions.  One  way  to  insure 
that  preclusion  is  to  withhold  any  feedback  to  the 
participants.  Preclusion  is  not  always  desirable  and 


feedback  is  often  necessary  to  achieve  an  objective, 
as  is  the  case  for  virtual  entity  insertion  into 
operational  displays,  with  realistic  response  lags. 
This  system,  however,  has  no  feedback  and  thus  no 
latency  requirement. 

Concent  Description 

The  description  will  be  short.  Experts  need  the 
flexibility  to  respond  with  their  best  design.  My 
knowledge  of  the  equipment  recommended  (some 
limited  and  some  extensive)  and  my  general  research 
indicates  that  the  proposal  is  feasible  within  a 
reasonable  cost  and  time. 

TSPI 

The  Precision  Lightweight  GPS  Receiver  (PLGR)  is 
capable  of  providing  suitable  position  and  time  data. 
It  is  a  differential,  precise-code,  three-pound  unit,  in 
service  for  nearly  five  years.  Tens  of  thousands  have 
been  delivered  worldwide.  There  are  several  versions 
of  the  PLGR  installed  on  military  groimd  vehicles 
and  used  as  hand  sets  by  infantry  units.  PLGRs  are 
used  operationally  and  thus  training  is  available  and 
provided. 

PLGRs  are  “authorized”  units,  which  indicates 
precise,  two-ffequency  accuracy,  free  from 
intentional  error  manipulation.  Differential  capable 
means  that  the  majority  of  bias  errors  can  be 
“subtracted”  from  the  position  solution  using  a 
differential  receiver,  located  on  a  surveyed  position 
within  about  200  miles.  Original  PLGR 
requirements  stated  the  ability  to  perform  the 
differential  correction  in  the  unit  itself  or  in  a  central 
location  after  transmission.  Both  methods  require 
knowledge  of  the  satellites  used  by  PLGR  to 
determine  position. 

If  the  Kalman  filter  (an  estimator),  pseudo-range 
measurement,  or  any  element  of  the  differential 
correction  procedures  restrict  centralized 
corrections,  then  the  transmission  of  differential 
receiver  calculations  must  be  made  to  each 
individual  PLGR.  That  will  complicate  but  not 
eliminate  the  qualities  of  this  proposed  concept. 

The  PLGR  was  designed  to  be  robust  and  it  appears 
to  be  so.  Anecdotal  evidence  indicates  that  the  wire 
connection  to  the  vehicular  antenna  may  be  a  weak 
element,  and  the  vehicular  antenna  itself  can  be 
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shielded  by  inappropriate  personnel  actions.  These 
shortcomings,  if  accurate,  can  be  overcome. 

The  PLGR  uses  RS-232  and  RS-422  data  ports. 

PLGR  obtains  time  from  the  GPS  satellites  with 
“atomic-accuracy”. 

GPS  satellites  provide  worldwide  coverage.  The 
goodness  (geometry  and  visibility)  of  the 
constellation  can  vary  with  earth  location  and  time, 
affecting  accuracy,  but  terrain  features  seldom 
prohibit  position  or  time  determination.  Some 
vegetation  and  urban  buildings  can  restrict  reception 
of  the  satellites  by  the  antenna  but  those  limitations 
may  not  be  cost-effective  to  overcome.  Detailed 
documentation  of  the  scope  of  the  limitation  should 
be  determined  and  reported. 

Data  Transmission 

The  objective  is  universal  transmission  without 
unique,  fixed  infrastructure.  That  can  be  achieved  by 
using  a  digital,  satellite-based,  cellular 
communications  system. 

Iridium  is  such  a  system  and  its  satellite  segment  is 
now  complete.  Other  systems  such  as  Globalstar  will 
soon  follow.  They  will  have  different  capabilities  for 
data  messaging,  error  correction,  modulation, 
frequency  use,  PLGR  interface,  reliability  and  cost, 
but  all  will  have  the  ability  to  transmit  numerous 
data  messages  containing  position,  time,  participant 


identification  and  GPS  satellite  numbers  from  a  cell 
phone  to  a  central  location  with  sufficient  update 
rates. 

That  central  location  should  contain  the  equipment 
necessary  for  differential  correction,  recording  and 
appropriate  display  of  TSPI.  It  should  be  portable 
and  movable  to  any  powered  location  hosting  a  test 
or  training  event.  Movement  should  not  require  great 
amounts  of  time,  money  or  equipment.  The  central 
location  should  also  control  participant  setup  and 
initiation. 

Cost 


My  cost  estimate  for  1000  cell  phone  attachments  to 
existing  PLGRs  is  about  one  to  two  million  dollars. 
Development,  test,  and  central  location  cost  might  be 
equal  to  that  amount.  Operation  and  maintenance  for 
a  year  should  be  also  be  in  the  one  to  two  million 
dollar  area.  Phone  charges  for  an  eight-hour  exercise 
with  a  thousand  participants  should  be  less  than  a 
hundred  thousand  dollars.  That  is  based  on 
continuous  operation  at  twenty  cents  a  minute,  a 
discount  for  volume  on  the  proposed,  published  rates 
per  minute. 

Conclusion 

An  instrumentation  system  suitable  for  use  by  testers 
and  trainers  during  a  single  event  is  feasible.  It 
should  provide  only  participant  position  and 
associated  time. 
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Abstract 

Advances  in  physics  based  computer 
modeling,  simulation,  and  visualization  are 
allowing  the  U.S.  Army's  Test  and  Evaluation 
Command  to  create  synthetic  environments 
for  a  wide  range  of  systems.  These  synthetic 
environments  are  being  used  to  exercise 
computer  models  of  new  weapons  systems  in 
order  to  plan  the  subsequent  "real-world" 
testing  in  an  actual  facility  at  a  test  range. 
Experience  is  showing  that  cost  effective  test 
programs  are  being  designed  with  greater 
emphasis  on  the  "computer  model  in  synthetic 
environment"  and  "hardware-in-the-loop" 
activities,  with  less  time  spent  in  a  "real- 
world"  test  range  or  facility  verifying 
predicted  outcomes.  Such  a  trend  has 
enormous  implications  for  decisions  on  where 
to  place  scarce  test  technology  resources  in 
the  21st  century.  This  paper  will  present  a 
coherent  strategy  for  prioritizing  investments 
in  synthetic  environments,  based  on 
developments  in  modeling  and  simulation, 
coupled  with  cost-benefit  analyses  of  the 
synthetic  versus  real-world  tradeoffs. 
Examples  will  be  presented  of  the  decisions 
being  made  today  that  affect  both  the 
management  of  U.S.  Army  test  ranges  and 
facilities,  and  the  investments  that  are  needed 
to  assure  a  viable  testing  capability  for  the  ^ 
21st  century. 


The  Test  and  Evaluation  (T&EI 
Environment 

At  least  for  the  past  five  years,  the  T&E 
community  has  experienced  an  environment 
of  decreasing  investment  dollars,  increasing 
complexity  of  weapons  systems,  and  greatly 
increasing  availability  of  computing  power. 
These  factors  have  led  to  investment 
strategies  that  focus  on  simulation-based 
testing  capabilities.  From  a  “facilities” 
standpoint,  the  shift  to  simulation-based 
testing  capabilities  has  profound  implications 
for  the  investment  strategy  of  the  future. 

An  Emerging  Strategy  in  1994 

At  the  end  of  the  Cold  War,  U.S.  defense 
policy  was  shifting  to  address  lower  intensity 
conflicts  with  less  sophisticated  adversaries. 
While  research  and  development  to  explore 
new  technologies  continued  at  a  fairly 
constant  level,  acquisition  of  new,  high 
technology  weapons  systems  began  to  come 
under  increased  scrutiny  from  a  cost 
standpoint.  Testing  and  evaluation  in  the 
U.S.  Army,  as  a  part  of  the  acquisition 
process,  began  to  experience  a  dovsmtum  in 
resources.  Evidence  of  the  lower  priority 
being  placed  on  investments  in  T&E  was  the 
complete  elimination  of  investment  funding 
for  major  Army  T&E  instrumentation  in  fiscal 
year  1994,  a  one-year  impact  of  over  $37M. 
Investments  in  all  Army  T&E  instrumentation 
dropped  from  a  high  of  SIOOM  in  fiscal  year 
1992,  to  $46M  in  fiscal  year  1994.  Overall, 
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the  iunding  for  Army  T&E  dropped  24% 
from  fiscal  year  1992  to  fiscal  year  1994. 

In  the  face  of  diminishing  resources,  the  real- 
world,  physical  testing  facilities  (such  as  road 
courses,  launch  complexes,  and  firing  ranges) 
continued  to  deteriorate.  Physical 
deterioration,  which  could  be  kept  in  check  at 
prior  levels  of  funding  through  preventive 
maintenance,  emergency  repair,  and  periodic 
refurbishment,  began  to  accelerate. 

At  the  same  time  that  fimding  was 
diminishing,  the  revolution  in  computing 
power  was  getting  underway.  At  the  personal 
computer  level,  the  “386”  processor  was 
replaced  by  the  “486”  in  1992,  with  the 
Pentium  processor  following  shortly 
thereafter.  Supercomputers  were  beginning  to 
demonstrate  the  capability  to  replicate 
accurately  and  cost  effectively  the  physical 
phenomena  associated  with  complex  events  in 
such  fields  as  computational  fluid  dynamics. 
The  realism  of  computer  generated  graphics 
was  increasing  quickly  with  increased 
computational  power.  High  data  rate 
networks  were  demonstrating  the  capability  to 
conduct  distributed,  interactive  simulations 
across  remote  locations.  Coupled  with  these 
advances  was  a  new  policy  in  the  Department 
of  the  Army  to  require  new  major  weapons 
systems  acquisition  programs  to  develop  and 
demonstrate  a  computer-based  simulation  of 
the  new  weapons  system  in  parallel  with 
developing  the  hardware  prototype. 

Computer  aided  design  and  computer  aided 
manufacturing  were  beginning  to  form  the 
basis  of  what  is  now  known  as  “simulation 
based  acquisition”  in  the  U.S.  Department  of 
Defense  (DoD).  It  became  evident  that  the 
road  ahead  in  military  acquisition  included 
exercising  computer  based  simulations  of  new 
hardware  in  a  realistic,  computer-based 
environment. 

All  these  capabilities  demonstrated  that  it  was 
becoming  feasible  to  consider  a  “continuum” 
of  testing  facilities.  The  beginning  of  the 
continuum,  starting  in  the  earliest  stages  of 
new  systems  development,  would  consist  of 


computer  models  of  the  new  weapon  being 
exercised  in  a  computer-generated 
environment.  For  example,  with  high 
technology  tools,  one  doesn’t  need  to  build  a 
hardware  prototype  of  a  new  truck  to  test 
whether  it  will  have  sufficient  ground 
clearance  over  rugged  terrain  when  fully 
loaded.  A  physics-based  model  of  the  truck, 
coupled  with  a  validated  terrain  model,  can  be 
used  to  explore  ground  clearances  under  a 
great  variety  of  conditions,  and  incremental 
re-designs  can  be  investigated  easily  without 
the  necessity  to  build  successive  hardware 
prototypes.  The  far  end  of  the  continuum 
would  be  a  real-world  test  of  a  hardware 
prototype  on  an  actual  test  track  late  in  the 
engineering  development  cycle.  Even  at  this 
“far”  end  of  the  continuum,  the  latest  version 
of  the  computer  model  of  the  hardware  could 
be  used  in  a  validated  computer-generated 
environment  to  predict  the  “edges”  of  the 
performance  envelope  that  needed  special 
attention  during  the  real-world  test. 

Considering  this  complex  environment  in 
1993  (diminishing  resources,  increasing 
simulation  capabilities,  deteriorating  test 
facilities,  and  increasing  complexity  of 
weapons  systems),  for  the  first  time  there 
appeared  to  be  an  opportunity  to  address  the 
critical  question  of  how  do  we  trade  off 
investments  in  simulation-based  testing  for 
investments  in  traditional,  real-world 
facilities.  And,  if  the  tester  were  to  consider  a 
trade-off  between  a  simulation-based  test 
capability  and  a  real-world  test  facility,  on 
what  basis  would  that  trade-off  be  made? 

Critical  Decision 

In  planning  for  the  fiscal  year  1995  budget, 
the  U.S.  Army  developmental  testing 
organization  made  the  critical  decision  to 
pursue  actively  a  comprehensive  set  of 
simulation-based  testing  capabilities.  This 
would  be  implemented  by  channeling 
investments  heretofore  targeted  toward  real- 
world  test  facilities  toward  simulation-based 
capabilities.  In  essence,  investment  funding 
would  be  diverted  to  “virtual  facilities”  from 
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real-world  facilities.  This  was  the  key 
decision  that  was  necessary  to  make  the 
development  of  simulation-based  testing  a 
reality. 

The  US  Army  developmental  testing 
organization,  known  as  Army  Test  and 
Evaluation  Command  (TECOM),  conducts  its 
technology  investment  program  by 
prioritizing  individual  projects  proposed  by 
the  six  test  centers.  Each  year,  the  test  centers 
(Aberdeen  Test  Center,  White  Sands  Missile 
Range,  Yuma  Proving  Ground,  Dugway 
Proving  Ground,  Redstone  Technical  Test 
Center,  and  Aviation  Technical  Test  Center) 
submit  project  descriptions  which  describe 
each  project  in  terms  of  the  capability  to  be 
provided,  the  amount  of  cost  avoidance 
expected  to  be  generated  by  the  project,  the 
safety  enhancements  to  be  achieved,  and  the 
relative  criticality  of  the  project  in  relation  to 
weapons  system  acquisition  programs.  The 
spending  plan  for  technology  investments  is 
developed  by  integrating  the  proposed 
projects  from  the  test  centers  into  a  prioritized 
list,  and  then  fiinding  the  investments  by 
beginning  at  the  top  of  the  list  and  ending 
with  the  lowest  priority  project  on  the  list  that 
can  be  accomplished  within  the  available 
investment  dollars. 

In  the  investment  planning  for  fiscal  year 
1995,  TECOM  stopped  the  normal 
prioritization  process  and  added  the  concept 
of  “supporting  a  virtual  proving  ground”  to 
the  decision  process.  The  test  centers  were 
given  the  opportunity  to  label  on-going 
projects  as  supportive  of  a  simulation-based 
testing  capability,  and  to  propose  new  projects 
that  would  support  such  a  capability.  Once 
identified,  the  Virtual  Proving  Ground  (VPG)- 
labeled  projects  would  have  added  priority  in 
the  funding  process.  The  test  centers 
responded  with  a  wide  array  of  simulation- 
based  test  capability  investments,  and  the 
Virtual  Proving  Ground  began  to  take  shape. 

In  fiscal  year  1995,  TECOM  committed  to 
spend  $8.4M  of  the  total  $26.8M 
instrumentation  budget  for  projects  supporting 


the  Virtual  Proving  Ground.  Priority  was 
given  to  projects  that  would  provide  a  return 
on  investment,  in  terms  of  saving  more  money 
for  the  test  customers  than  it  cost  to  build  the 
project.  Test  centers  began  to  realize  that 
future  investment  funding  for  their  test  center 
would  depend  to  a  large  extent  on  the  degree 
to  which  they  could  propose  to  build  valid, 
cost-saving  simulation-based  test  capabilities. 
The  $8.4M  spent  in  fiscal  year  1995  contrasts 
sharply  with  the  total  of  $4.5M  for 
simulation-based  testing  capabilities  that  were 
identified  before  the  priority  scheme  was  put 
into  place.  Simulation-based  testing 
capabilities  became  the  “facilities”  of  the 
future,  beginning  to  displace  investments  in 
conventional  facilities  such  as  firing  ranges 
and  test  tracks. 

The  Plan 

In  the  planning  for  investments  to  be  made  in 
fiscal  year  1996,  TECOM  realized  that  the 
concept  of  simulation-based  testing  facilities 
needed  a  long-term  framework.  A  Virtual 
Proving  Ground  Working  Group  was 
established,  using  experts  from  each  of  the 
TECOM  test  centers,  with  the  goal  of 
producing  a  strategic  plan  to  guide 
investments  in  simulation-based  testing.  The 
Working  Group  defined  four  categories  that 
would  be  used  to  partition  new  investments  in 
the  VPG: 

Category  A  -  Projects  needed  to  develop  a 
near  term  capability  (by  fiscal  year  1998) 
for  the  Comanche  (the  scout  /  attack 
helicopter),  the  Javelin  (a  man-portable 
anti-tank  weapon),  the  Crusader  (the  next 
generation  self-propelled  howitzer)  or  the 
THAAD  (Theater  Hgh  Altitude  Air 
Defense  missile  system). 

Category  B  -  Projects  needed  at  all 
TECOM  test  centers  to  support  simulation 
based  testing  capabilities.  These  were 
projects  such  as  common  simulation  tools, 
data  transmission,  data  reduction,  and 
networking. 
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Category  C  -  Projects  at  specific  TECOM 
test  centers  other  than  the  above  in  support 
of  Army  test  programs. 

Category  D  -  Projects  in  support  of 
simulation  based  testing  capabilities  being 
developed  by  all  three  Services  (Army  in 
conjunction  with  Navy  and  Air  Force). 

Using  these  categories,  TECOM  was  able  to 
determine  the  relative  contribution  of  the 
various  projects  in  relation  to  the  available 
resources  for  fiscal  year  1996.  TECOM 
funded  all  Category  A  projects  for  fiscal  year 
1996,  and  some  of  the  Category  B  projects, 
totaling  $15.2M.  However,  the  funding  was 
insufficient  to  purchase  all  tools  needed 
across  all  test  centers,  and  there  was  no 
money  available  for  the  installation-specific 
projects  or  the  joint  service  projects. 

In  addition  to  the  project-specific  resources 
planning  for  fiscal  year  1996,  TECOM  was 
developing  the  second  draft  of  the  Modeling 
and  Simulation  Master  Plan.  The  goal  of  this 
plan  was  to  achieve  TECOM’ s  future  testing 
mission  in  the  “synthetic”  world.  TECOM 
would  no  longer  be  confined  to  the  test-fix- 
test  mode  of  engineering  development,  but 
would  elevate  to  a  model-simulate-test-model 
mode.  The  Modeling  and  Simulation  Master 
Plan  set  forth  a  structure  for  this  new  testing 
mission,  and  a  time-line  for  actions  necessary 
to  achieve  the  structure.  The  critical  benefit 
of  this  Plan  was  the  ability  for  all  TECOM 
staff  to  comprehend  the  scope  and  direction  of 
the  initiative  to  move  into  synthetic  testing; 
each  element  of  the  staff  could  interpret  the 
initiative  in  the  context  of  their 
responsibilities. 

The  only  problem  at  this  point  was  that  the 
Army’s  developmental  testing  funding  had 
just  experienced  another  major  decrease  in 
conjunction  with  overall  defense  budget 
cutting,  and  simulation-based  testing  was 
moving  out  of  reach  at  the  very  time  that  the 
structure  was  being  defined. 


The  Resources 

In  early  1996,  TECOM  had  developed  a 
detailed  plan  of  the  funding  requirements  for 
simulation-based  facilities  for  fiscal  year  1997 
and  beyond.  Requirements  in  fiscal  year  1997 
were  $14.3M  alone.  In  the  DoD  budget 
structure,  TECOM  is  on  a  partial  “trading 
basis,”  having  to  “earn”  the  direct  costs  of 
testing  from  test  customers,  and  being 
provided  an  ever-decreasing  yearly  stipend 
for  overhead  expenses.  The  funds  for  direct 
testing  costs  are  given  to  the  weapons  systems 
project  managers  as  part  of  the  overall 
weapons  system  project  funding.  The  project 
managers  can  then  decide  to  get  testing 
accomplished  either  at  in-house  (TECOM) 
facilities  or  at  private  sector  facilities. 

Although  the  funds  for  direct  test  costs  were 
in  the  hands  of  the  project  managers,  the 
overhead  account  allocated  to  TECOM  was 
not  able  to  keep  up  with  modernization 
investments  needed  to  build  the  simulation- 
based  testing  capabilities. 

TECOM  developed  a  plan  to  articulate  the 
following  logical  argument: 

If  simulation-based  test  facilities  are  more 
efficient  than  “brick  and  mortar”  test 
facilities  in  providing  information  to  test 
customers,  then  the  direct  testing  costs 
borne  by  the  project  managers  should  go 
down. 

If  the  aggregate  of  all  efficiencies  realized 
in  the  hands  of  the  project  managers 
exceeds  a  proposed  increase  in  funding  (in 
the  overhead  account)  to  purchase  these 
new  facilities,  then  the  DoD  has  realized  a 
substantial,  positive  cost  avoidance. 

Through  a  detailed  analysis  of  benefits  to  be 
gained  by  each  element  of  the  simulation- 
based  testing  capabilities,  TECOM  was  able 
to  convince  the  DoD  budget  planners  that  a 
modest  investment  in  simulation-based 
facilities  would  more  than  justify  the  expense 
of  acquiring  such  facilities.  In  gross  numbers. 
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the  simulation-based  facilities  would  cost 
roughly  $77.5M  over  a  six  year  period,  with 
total  cost  avoidances  exceeding  $150M.  The 
funding  to  achieve  the  simulation  based 
testing  capabilities  was  to  come  from  two 
sources:  1)  a  portion  of  the  previously 
allocated  investment  budget  of  TECOM,  and 
2)  a  new  budget  allocation  based  on  the  cost 
avoidance  to  be  realized  from  simulation 
based  testing.  The  detailed  cost  profile  (in 
millions  of  dollars)  was  as  follows: 

Funds  previously  approved: 

FY98  FY99  FYOO  FYOl  FY02  FY03  Total 


6.3  7.2 

3.3 

2.7 

3.3 

3.3 

26.1 

New  funds: 
9.7  8.5 

8.7 

8.4 

7.8 

8.3 

51.4 

Total  cost: 
16.0  15.7 

12.0 

11.1 

11.1 

11.6 

77.5 

What’s  on  the  Ground 

Prior  to  the  FY98  program  to  fund  new 
simulation-based  testing  capabilities,  a 
number  of  on-going  projects  were  forming  the 
basis  for  gaining  confidence  in  the  application 
of  computer  simulations  to  developmental 
testing.  Examples  of  these  projects  are  as 
follows: 

Aberdeen  Test  Center  -  The  Fire  Control 
Test  Complex  at  Aberdeen  Test  Center 
uses  computer  guided  lasers  to  generate 
target  presentations  on  the  inside  surface  of 
a  100-foot-radius  hemisphere.  Full  scale 
weapons  platforms  with  integrated  fire 
control  systems  (for  example,  the  Ml  A2 
Abrams  tank  and  the  AH-1  Cobra 
helicopter)  are  placed  inside  the  dome  and 
the  capabilities  of  their  fire  control  systems 
are  tested  with  soldiers  operating  the 
weapons  systems.  Tracking  errors  can  be 
recorded  and  analyzed  to  form  a  basis  for 
improvements  in  tracking  software  and 
hardware. 

White  Sands  Missile  Range  -  The  Smart 
Munitions  Test  Suite  at  White  Sands  has 
integrated  a  super  computer  with  the  latest 


multiple  object  tracking  radar  to 
demonstrate  the  capability  to  "aim"  camera 
coverage  of  multiple,  high  speed  events 
associated  with  submunitions  dispensed 
from  high  speed  rockets  and  missiles. 
Typically,  these  submunitions 
autonomously  detect,  identify  and  track 
ground  targets.  Not  only  can  the  Smart 
Munitions  Test  Suite  track  and  capture  all 
these  events,  it  can  also  produce  a 
computer  simulated  visualization  of  the 
planned  event  with  sufficient  accuracy  to 
allow  testers  to  optimize  instrumentation 
locations  and  target  complexity.  This  test 
planning  capability  gives  the  Smart 
Munitions  Test  Suite  the  advantage  of 
being  able  to  assist  in  optimizing  these 
extremely  expensive  tests.  The  $5M  initial 
investment  in  this  virtual  firing  range 
generates  more  than  $5M  cost  avoidance 
annually. 

Redstone  Technical  Test  Center  -  The 
Simulation  /  Test  Acceptance  Facility  at 
the  Redstone  Technical  Test  Center 
performs  lot  acceptance  testing  on  Hellfire 
missile  production  lots  without  actually 
firing  the  missile.  The  missile  is  an  air-to- 
ground,  radar  guided  missile  fired  from  the 
Apache  helicopter.  In  this  facility,  a  fully 
capable,  “live”  missile  is  placed  in  a  test 
fixture  where  it  is  free  to  move  through  six 
degrees  of  freedom.  Target  signatures  are 
presented  to  the  radar  seeker,  which  in  turn 
generates  commands  to  the  missile’s 
control  surfaces.  As  the  control  surfaces 
move,  the  test  fixture  computes  the 
resulting  dynamic  forces  on  the  missile 
airframe  and  translates  these  into 
accelerations  and  displacements  of  the  test 
fixture.  Simultaneously,  the  target 
signatures  being  presented  to  the  radar 
seeker  change  position  corresponding  to 
the  seeker’s  movements  as  a  part  of  the 
missile  airframe.  In  effect,  the  missile 
“flies”  to  the  target  with  all  of  the 
guidance,  control,  and  dynamic  functions 
performed  exactly  as  in  a  real  flight.  The 
only  functions  not  performed  are  the  rocket 
motor  bum  and  the  warhead  explosion. 
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If  all  functions  are  nominal,  the  missile  is 
returned  to  the  stockpile  as  a  folly  capable 
round,  rather  than  having  been  expended  in 
a  live  firing.  The  initial  investment  of  $6M 
for  this  facility  returns  over  $8M  per  year 
in  cost  avoidance  in  terms  of  missiles  that 
would  have  been  expended,  but  are  rather 
returned  to  the  stockpile  as  folly  capable. 

Testing  and  Science  &  Technology  (S&TI 
Partnership  -  TECOM  is  partnering  with 
the  Army  Research  Laboratory  and  the 
University  of  Arizona  to  develop  the 
Hardened  Subminiature  Telemetry  and 
Sensor  System  (HSTSS)  which  is  a 
physics-based  model  for  examining 
munition  and  weapon  system  performance. 
The  HSTSS  is  a  new  instrumentation 
package  that  fits  into  the  base  of  an 
artillery  or  tank  projectile,  and  transmits 
data  on  the  accelerations  being  experienced 
by  the  projectile  while  in  flight.  These 
data  can  be  translated  into  flight  attitudes 
of  the  projectile,  which  indicate  whether 
the  projectile  is  flying  “on-course.”  The 
HSTSS  will  be  used  to  examine  the 
expected  performance  characteristics  of 
tank  and  artillery  projectiles,  as  well  as  aid 
in  data  analysis  of  the  Crusader  test 
program.  Performance  will  be  predicted 
through  the  modeling  of  the  new  charge 
system  and  the  30  combinations  of  "fose- 
to-projectile." 

Dynamic  Infrared  Scene  Projector 
IDIRSP^  -  TECOM  is  developing  the 
DIRSP  to  provide  real-time,  dynamic, 
complex,  in-band  infrared  (IR)  energy  into 
the  entrance  aperture  of  the  Army’s  2"“* 
generation  forward  looking  infrared 
(FLIR)  and  missile  IR  seeker  sensor 
subsystems  in  a  laboratory.  The  DIRSP 
presents  the  target  scene  as  an  infrared 
“picture”  varying  in  real-time  with  the 
attitude  of  the  missile  as  it  flies  to  the 
target.  The  DIRSP  will  initially  support 
the  Improved  Target  Acquisition  System, 
Improved  Bradley  Acquisition  Subsystem, 
Javelin  missile  IR  seeker  subsystem,  and 


the  Comanche  Electro-Optical  Sensor 
System. 

Virtual  Range  -  TECOM  formed  a  team 
with  members  from  several  Army 
organizations  to  develop  unique 
procedures  to  collect  multispectral 
calibrated  target  and  background 
signatures,  integrated  with  range  and  target 
characterization  databases,  into  a  virtual 
synthetic  scene  suitable  for  use  with  a 
number  of  simulation  applications.  End 
capabilities  will  consist  of  a  core  set  of  3- 
dimensional  infrared  target  and 
background  models  representing  the  desert, 
arctic,  temperate,  and  tropical  climates. 

The  end  product  of  these  efforts  will 
directly  support  the  Javelin/Follow-on-to- 
TOW  development  and  performance 
assessments. 

TECOM  anticipated  the  need  to  be  able  to 
validate  the  efficiencies  being  realized 
throughout  the  acquisition  of  simulation  based 
testing  capabilities.  One  could  reasonably 
expect  that  the  Department  of  Army  would  be 
very  interested  in  seeing  that  over  $77M  in 
investments  was  being  put  to  use  for  testing 
capabilities  that  meet  the  customer’s 
requirements  with  increasing  efficiencies. 

That  is  why  the  plan  for  the  Virtual  Proving 
Ground  included  a  timetable  for  assessing  the 
Early  User  Utility  of  these  simulation  based 
testing  capabilities.  Simulation  based 
facilities  to  be  acquired  over  the  planned 
period  are  prioritized  so  that  projects  with  a 
large  cost  avoidance  in  relation  to  investment 
cost,  coupled  with  a  relatively  short 
acquisition  time,  will  be  completed  early  in 
the  overall  program.  To  the  extent  that  these 
projects  can  be  identified,  fonded  and 
completed,  this  approach  assures  that  the 
fonding  allocation  can  be  “protected” 
throughout  the  project  based  on  early, 
successful  elements. 

Beginning  this  summer.  Early  User  Utility 
assessments  of  the  Virtual  Proving  Ground 
will  be  performed  at  TECOM  test  centers  and 
contractor  facilities.  To  assure  objectivity. 
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these  assessments  will  be  conducted  by  an 
Army  systems  analysis  organization 
independent  of  TECOM.  Actual,  working 
simulation-based  testing  capabilities  will  be 
assessed  for  their  utility  at  the  following 
locations: 

Sikorsky  Aircraft  Corporation  -  TECOM 
has  partnered  with  the  project  manager  for 
the  Army’s  Comanche  helicopter  program 
to  build  the  Flight  Test  Support  Simulator 
(FTSS)  at  the  manufacturer’s  facility.  This 
system  will  allow  pilots  and  aircraft 
engineers  to  design  the  test  program  for  the 
actual  flight  article,  test  weapons  system 
engagements  via  the  infrared  weapons 
guidance  and  control  systems,  investigate 
flying  qualities  under  adverse  conditions, 
and  evaluate  human  factors  and  subsystem 
integration. 

JAVELIN  /  Follow-on-to-TOW  -  TECOM 
is  working  as  a  part  of  the  Follow-on-to- 
TOW  acquisition  team  to  develop  an 
extensive  Simulation  Test  and  Evaluation 
Plan  to  support  the  program  from  early 
development  throughout  the  life  cycle. 

The  simulation-based  testing  capabilities 
will  include  a  combination  of  fully 
computer-based  test  simulations  as  well  as 
hardware-in-the-loop  simulations.  The 
involvement  of  TECOM  in  this  acquisition 
program  will  serve  as  a  “pathfinder”  to 
demonstrate  how  fijture  acquisition 
programs  can  integrate  modeling  and 
simulation  early  into  the  development 
process  to  reduce  cost,  risk  and 
development  time. 

Facilities  Decisions  for  the  Future 

The  emerging  experience  with  simulation- 
based  testing  capabilities  has  profound 
implications  for  test  facilities  of  the  future. 
Experience  is  showing  that  a  plan  to  shift 
toward  agile,  re-configurable  simulation- 
based  test  capabilities  can  compete 
successfully  with  conventional  investments  in 
traditional  launch  complexes,  road  courses, 
and  firing  ranges. 


An  environment  of  either  constant  or 
decreasing  resources,  as  is  likely  in  the  next 
five  to  ten  years,  will  continue  to  exert 
pressure  for  more  cost  avoidance  and 
efficiency  gain  for  each  unit  of  investment  in 
new  facilities. 

Coupled  with  the  need  for  investment 
efficiency  is  the  recognition  that  increasingly 
complex  weapons  systems  are  driving  up  the 
“cost  per  test  event.”  This  has  been  evident  in 
missile  systems  and  aircraft  for  some  time. 
However,  with  the  advent  of  “brilliant” 
submunitions  and  projectiles,  the  “cost  per 
test  event”  for  previously  considered 
“conventional”  weapons  testing  (tank  main 
armament  and  anti-tank  weapons,  for 
example)  are  becoming  prohibitively 
expensive.  Again,  this  cost  per  test  event  in 
real-world  testing  applies  pressure  toward 
developing  simulation-based  testing 
capabilities  where  literally  hundreds  to 
thousands  of  test  events  can  be  simulated  for 
the  expense  of  one  live  shot. 

Needless  to  say,  the  advances  in  computer 
technology  (including  hardware,  software  and 
networking  technologies)  are  bringing  the 
objective  of  valid,  physics-based  simulations 
closer  to  reality.  Energetic  events  such  as 
detonation  of  high  explosives  can  be 
visualized  and  studied  using  computational 
fluid  dynamics.  In  designing  their  latest 
vehicle  engine,  Chrysler  Corporation  used 
computer  simulations  to  investigate  the  sound 
of  over  two  hundred  intake  manifold  designs 
before  ever  casting  the  first  article.  (And  this 
was  done  with  technologies  available  over 
two  years  ago.)  Advances  in  computer  aided 
design  and  computer  aided  manufacturing  are 
assuring  that  simulation-ready,  computer- 
based  prototypes  will  be  produced  as  a  matter 
of  course  in  the  development  of  new  military 
systems. 

All  of  these  vectors  point  toward  increased 
movement  in  the  direction  of  increased 
investments  in  simulation  based  testing 
capabilities. 
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Certainly,  real-world  test  facilities  have  a 
place  in  the  overall  future  testing 
infrastructure.  Real-world  tests  will  continue 
to  be  needed  in  the  mid-future  to  validate  the 
“edges”  of  the  performance  envelope,  and  to 
confirm  that  the  physics-based  models  can 
predict  accurately  the  real-world  performance. 
However,  it  is  no  longer  the  case  that  one  can 
ask  whether  a  real-world  test  capability  can  be 
replaced  at  least  in  part  by  a  simulation-based 
test  facility.  It  is  becoming  a  question  of 
“when.” 
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Abstract 

With  the  increasing  sophistication  of  new 
weapon  systems  the  overriding  requirement 
for  aerial  targets,  is  that  they  should  be 
realistic:  i.e.  they  should  have  the  same 
signature,  size,  shape  and  performance  of  the 
threats  that  they  are  replicating. 

In  many  cases  the  only  way  all  the  above 
criteria  can  be  met  simultaneously  is  by 
effectively  replicating  the  threat  itself,  for 
example  the  use  of  full  scale  drones.  This 
approach  although  very  thorough  is  rather 
expensive.  If  however  only  a  limited  number 
of  target  characteristics  are  required  it 
becomes  possible  to  use  cheaper  alternatives, 
such  as  enhanced  sub  scale  drones  and/or 
towed  targets.  The  types  of  enhancement 
required  Include  thermal  and  radar  cross 
section  augmentation. 

The  signatures  of  the  present  generation  of 
sub  scale  drones  and  towed  targets,  both  IR 
and  radar  augmented  versions,  are 
unrepresentative  of  the  threats  that  are  now 
being  considered  by  weapon  system 
designers.  This  short  coming  is  particularly 
prominent  when  weapons  fitted  with  IR 
imaging  seekers  are  being  tested,  because 
these  targets  have  reduced  spatial  dimensions 
and,  in  the  case  of  towed  targets,  consist  of  a 
single  point  source  of  radiation.  The  seeker 
could  identify  this  small  point  as  a  decoy  and 
ignore  it,  and  start  searching  for  another 
target,  possibly  finding  the  towing  aircraft.  A 
similar  situation  exists  for  weapons  fitted  with 
radar  seekers,  which  may  look  for  effects  such 
as  scintillation,  glint  and  engine  modulation 


rather  than  just  the  largest  return. 

This  paper  describes  the  various  augmentation 
techniques  available  and  imder  development 
by  Airborne  Instrumentation,  T&E  Ranges 
Sector,  DERA  Aberporth,  and  WX3, 
Weapons  Sector,  DERA  Fort  Halstead,  to 
enhance  sub  scale  drone  and  towed  targets  in 
an  attempt  to  provide  realistic  represeritations 
of  typical  threats. 

Introduction 

Airborne  Instrumentation  (AI)  is  a 
Department  within  the  Test  and  Evaluation 
Ranges  Sector  of  the  Defence  Evaluation  and 
Research  Agency  (DERA)  responsible  for  the 
enhancement  and  augmentation  of  target 
system,  primarily  aerial  but  also  to  a  lesser 
extent  sea  and  land.  Some  of  the  areas 
currently  under  investigation  include;  GPS 
and  GPS/INS  for  both  position  recording  and 
control,  miss  distance  seorers  both  scalar  and 
vector  and,  the  subject  of  this  paper,  IR  (in 
collaboration  with  WX3)  and  radar 
augmentation. 

AI  is  based  at  the  DERA  Aberporth  Range,  on 
the  West  Wales  coast  and  provides,  a  service 
to  several  DERA  Ranges  distributed  across 
the  UK  as  well  as  to  Ministry  of  Defence 
(MoD)  and  external  customers. 

With  advances  in  modem  weapon  seeker 
technology  it  is  becoming  more  and  more 
important  to  provide  targets  that  accurately 
represent  the  signatures  of  tme  threats.  This 
requirement  exists  equally  for  Infra-Red  (IR) 
and  Radar  seekers.  This  paper  has  been 
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written  to  give  an  insight  into  the  problems 
associated  with  achieving  this  and  some  of  the 
solutions  currently  being  investigated  by  AI 
and  WX3. 

Infra-red 
The  Requirement 

Aerial  targets  are  required  to  fulfil  a  number 
of  uses  -  for  missile  practice,  gunnery 
practice,  missile  development,  warning 
system  development,  and  many  other 
applications. 

The  constantly  increasing  levels  of 
sophistication  in  the  design  of  weapons  and 
other  systems  for  air  attack  and  defence 
requires  a  variety  of  targets  for  the  safe  testing 
of  prototype  devices  and  for  weapons 
practice.  Targets  are  not  only  being  enhanced 
using  on-board  devices  to  increase  their  own 
infrared  signature  but  with  current  and  future 
missile  systems  having  increasingly  more 
advanced  counter-countermeasure  algorithms 
to  reject  expendable  IR  decoys,  it  is  also 
necessary  to  enhance  the  aerial  targets  with 
flare  dispensers. 

To  a  certain  extent  the  development  of  air 
targets  is  reactive  rather  than  pro-active  (as 
with  countermeasure  development),  especially 
in  the  case  of  missile  seekers,  where  tracking 
and  acquisition  algorithms  may  be  sensitive, 
and  information  regarding  what  may  be 
suitable  as  a  target  is  not  readily  available. 

In  addressing  the  area  of  IR  enhancement  of 
aerial  targets  several  points  need  to  be  taken 
into  account.  Firstly  the  technique  to  be  used 
in  obtaining  radiant  emission  -  for  instance 
whether  an  energetic  material  is  more  suitable 
than  electric  means.  Secondly  whether  the 
target  is  required  for  a  single  presentation  or 
for  multiple  presentations.  Thirdly  the  type  of 
materials  to  be  used  for  heating  -  there  are  a 
multitude  of  pyrotechnic  compositions  as  well 
as  other  energetic  materials  available. 
Fourthly  the  type  of  emission  required  -  is  a 
hot  point  source  or  a  large  area  source 
required,  or  just  an  increase  of  a  few  10s  of 


degrees  over  a  small  area  of  the  target? 
Finally  one  needs  to  consider  enhancing  a 
target  by  using  expendable  flares,  which 
greatly  increases  the  usefulness  of  the  target 
in  testing  seeker  CCM  systems. 

Even  with  an  initial  design/requirement  in 
place  there  are  many  problems  associated  with 
the  development  of  an  IR  enhanced  target, 
especially  windspeed  that  can  have  a  drastic 
effect  on  IR  emission. 

The  remainder  of  this  section  will  address 
these  points  and  discuss  their  association  and 
impact  in  aerial  target  IR  augmentation 
design. 

Solutions 

IR  Emission  Enhancement  Techniques 
With  towed  targets  having  no  engines  and 
most  model  drones  having  smaller  signatures 
than  a  realistic  target,  all  of  the  towed  targets, 
piston  engine  drones,  and  some  of  the  turbojet 
drones  need  enhancing  to  give  a  more 
representative  signature. 

There  are  two  main  means  of  providing  IR 
augmentation  or  enhancement,  by  use  of 
energetic  materials  and  by  electrical  means 
from  heaters.  Both  are  discussed  further 
below. 

Energetic  materials  There  are  several 
different  energetic  materials  used  for  target 
enhancement  and  each  has  distinct 
characteristics  that  make  it  suitable  to  specific 
types  of  augmentation,  the  following  table 
gives  examples  of  some  of  the  more  common 
materials  and  possible  applications  where  they 
might  be  used. 
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Family 

Material 

Charactcrislic.s 

Pi  iinary  Use 

Flammable  & 

Pyrophoric 

liquids 

Hydrocarbon 

burners 

IR  emission  from 
heated  gauze 

iinlianccr 
no  plume 

Alkylaluminiums 

Specific  ill  cmiiier  cf. 
Kerosene 

iinhancer 

Uxtended 

plume 

Pyrotechnics 

MTV 

Grey  body  emitter 

2000‘'C 

llnhnneer  or 
cxpendiibie 
(IRCM) 

Specific  emitters 

Spectral  enhancement 
in  the  vis/near  IR 

Unluincci 
for  specific 
i:0  .SYiticms 

(‘■■■8 

trackers) 

Thermites 

Grey  body  emission  • 
no  flame  plume 

Enhancer/ 

Expendable 

Spark  generators 

Large  area  of  IR 
emission 

Enhancer/ 

Expendable 

Pyrotechnic 
/propellant  mixes 

Specific  IR  emitters* 
lower  energy  than  MTV 

Ex|}ondahlc 

Table  2.  Emissive  Materials,  their 
Characteristics  and  Possible  Uses. 


From  the  table  it  can  be  seen  that  there  are 
two  distinct  families  of  energetic  materials; 
combustible  liquids  and  pyrotechnic 
materials.  These  are  discussed  in  more  detail 
in  the  following  sections. 

Combustible  Liquids  The  advantages  of 
using  liquid  systems  for  IR  sources  are  that 
they  can  be  turned  on  and  off  as  required, 
whereas  a  pyrotechnic  is  a  once  only  event. 
Flammable  liquids  offer  a  means  of  obtaining 
more  spectrally  selective  emissive  source  by 
either  the  primary  emission  from  the  reaction 
products  or  by  secondary  emission  from  a 
gauze.  The  liquids  may  be  split  into  two  types, 
those  that  are  flammable  only  with  a  source  of 
ignition  and  those  that  are  spontaneously 
flammable  on  contact  with  air.  These  latter 
pyrophoric  liquids  have  been  discussed  and 
tested  previously  for  target  applications  and 
will  not  be  expanded  upon  here. 

Liquids  have  the  advantage  of  giving  specific 
emission  and  energy  on  tap,  although  have  the 
disadvantages  of  lower  densities  (less  energy 
available  per  unit  mass)  than  solid  pyrotechnic 
mixes  and  susceptibility  to  fast  airstreams. 
Liquid  combustibles  are  commonly  used  for 
IR  augmentation  as  fuel  for  plumers  and 
burners.  Plumers  and  burners  in  current  use 
are  fuelled  by  propane  gas  or  by  jet  fuel 
supplied  from  the  drones  fuel  tank  or  from  an 
extra  tank  in  the  fuselage.  The  plumers  use 


mainly  jet  fuel  and  produces  a  flame  out  of 
the  rear  of  targets  or  out  of  wing-tip  mounted 
pods  fitted  on  target  drones.  Burners  generally 
use  propane  gas  that  is  used  to  heat  up  a  metal 
plate  or  gauze  to  produce  a  hot  metal  emitter. 
The  plumers  can  be  further  enhanced  by  using 
an  IR  Pyrotechnic  slurry  containing,  for 
example  an  ionic  oxidant,  an  organic  solvent 
and  a  metal  fuel  such  as  magnesium. 

Pyrotechnic  Materials  The  most  common 
method  of  pyrotechnic  enhancement  used  is 
by  addition  of  flares,  particularly  on  the  piston 
engine  targets,  which  have  a  very  low 
signature.  Flares  produce  hot  sources  and  in 
towed  targets  they  generally  have  a  larger 
signature  than  the  towing  aircraft.  An 
alternative  method  is  to  use  a  thermite 
composition  in  a  crucible.  The  thermite  is 
ignited  and  heats  the  crucible  in  which  it  is 
enclosed,  this  will  produce  a  hot  source  with 
little  or  no  flame  apparent  to  an  external 
observer.  These  pyrotechnic  systems  are 
discussed  later. 

Pyrotechnics  radiate  energy  in  all  wavebands 
throughout  the  UV  and  IR  spectral  regions 
depending  on  the  nature  of  the  compositions. 

Pyrotechnic  Thermites  Thermite  materials 
have  been  used  extensively  in  the  pyrotechnic 
industry  as  a  method  of  producing  ceramics 
and  hot  solid  reaction  product  without 
generating  many  gaseous  species.  The  target 
application  is  mainly  concerned  with  using 
thermite  material  to  act  as  large  thermally 
emissive  beacons  for  towed  targets  and 
missile  tracking  beacons. 

The  characteristics  of  thermite  reactions  are 
that;  they  produce  small  quantities  of  gaseous 
products;  they  can  exhibit  high  reaction 
temperatures  -  up  to  3000  °C;  they  produce 
molten  or  solid  slag;  they  are  often  difficult  to 
ignite,  with  ignition  temperatures  sometimes 
above  800  °C;  and  they  generally  have  a  high 
density,  due  to  the  metal  oxides  used.  As  a 
means  to  provide  a  hot  emitting  area  they  are 
very  useful  although  they  need  to  be  encased 
in  a  suitable  container  with  regard  to  the 
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temperature  of  reaction  of  the  thermite,  i.e.  the 
melting  point  of  steel  is  around  1500  °C 
which  limits  some  of  the  thermites  that  can  be 
used.  The  emission  from  the  device  is 
dependent  on  the  heated  area  of  casing 
material,  the  emissivity  of  the  surface  (this 
can  change  if  oxidation  occurs  at  high 
temperature),  and  the  heat  flow  characteristics 
of  the  container  (thickness  and  mass  of  metal 
etc). 

The  thermites  also  find  use  in  pyrotechnic 
batteries  where  they  ignite  to  melt  the 
electrolyte  and  provide  electrical  power. 
These  batteries  get  very  hot  themselves 
(several  hundreds  of  degrees)  and  could  be 
used  as  a  power  source  for  other  electrical 
means  of  heating. 

Spark  Generators  Some  pyrotechnics 
produce  high  melting  point,  solid,  reaction 
products.  Titanium  is  an  example  producing 
large  particles  of  titanium  oxide  in  the 
pyrotechnic  plume.  The  higher  heat  capacity 
of  these  particles  compared  with  vapour  phase 
reaction  products  result  in  a  long  low 
temperature  plume  from  the  target  which, 
although  low  in  total  energy,  can  be  easily 
observed  with  thermal  imaging  equipment 

Expendable  devices  The  use  of  targets  for 
missile  testing  requires  the  use  of  expendable 
IR  countermeasures  to  realistic  test  seeker 
counter  countermeasure  algorithms.  The  most 
common  stores  used  for  this  are  all  based  on 
MTV  composition. 

Experimental  payloads  can  be  designed  for  all 
of  these  countermeasure  cartridges  so  that  the 
total  energy,  the  distribution  of  energy 
spectrally  and  spatially,  and  the  bum  time  of 
the  expendable  store  can  all  be  tuned  to  any 
particular  target  requirement. 

Electrical  Depending  on  the  power  available, 
an  electrical  heater  element  or  bulb  can  be 
used  for  an  IR  source.  Xenon  lamps  can  be 
used  as  a  source  of  near  IR  energy  and  could 
be  focused  to  improve  their  irradiance  in  a 
particular  direction. 


The  use  of  electric  means  is  limited  to  low 
levels  of  heating  as  power  is  always  limited 
and  the  cooling  by  windspeed  very  efficient. 

Problem  Areas  regarding  IR  Enhancement 

Static  /  dynamic  performance  One  of  the 
biggest  problems  regarding  the  design  and 
development  of  IR  sources  for  aerial  targets  is 
the  difference  that  occurs  in  infrared  emission 
between  a  static  device  and  a  device  subjected 
to  high  windspeeds.  Windspeed  is  incredibly 
effective  at  cooling  heated  surfaces  and  the 
hot  exhaust  products  from  pyrotechnic  flames 
and  hydrocarbon.  This  is  not  a  matter  of  a 
losing  10%  or  so  of  the  energy  but  reducing 
the  energy  of  emission  by  a  factor  of  8  or  so. 
For  instance  a  flare  for  a  towed  target  might 
give  out  a  couple  of  kW  sr-1  in  the  IR  region 
when  burnt  statically  but  this  could  be 
reduced  to  a  few  hundreds  of  W  sr-1  when 
combusting  in  a  300  knot  airstream.  This  is 
due  to  the  rapid  cooling  of  the  hot  reaction 
products  by  the  airflow. 

Another  problem  caused  by  the  airflow  is  the 
restriction  on  the  physical  area  of  emission 
from  a  plume  or  pyrotechnic  flame.  Under 
static  air  conditions  a  pyrotechnic  can  be 
designed  to  emit  hot  material  over  a  large 
area,  but  in  a  windspeed  the  reaction  products 
are  swept  away  with  the  airflow  around  the 
target. 

Wind  tunnel  testing  It  -is  possible  to  carry  out 
wind  tunnel  trials  to  assess  this  windspeed 
effect.  DERA  has  a  facility  suitable  for  the 
safe  firing  of  combustive  materials.  It  is  based 
at  DERA  at  Boscombe  Down  and  is  capable 
of  windspeeds  of  up  to  300  knots. 

Wind  tunnels  can  also  highlight  problems 
with  design  associated  with  the  plume  from  a 
target.  The  re-circulation  of  air  around  the 
back  of  a  target  where  flares  or  other  IR 
augmentation  devices  are  often  placed  can 
cause  hot  plume  material  to  damage  the 
surrounding  fabric  of  the  target  and  can  even 
cause  parasitic  ignition  of  other  flares. 
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Summary 

Depending  on  the  requirement  it  is  possible, 
using  a  combination  of  the  techniques 
discussed,  to  provide  suitable  IR  target 
augmentation  and  expendable  flares,  for  CCM 
evaluation.  However  one  requirement  that  can 
not  be  fully  met  by  the  means  discussed  is  the 
evaluation  of  imaging  seekers.  To  fully 
evaluate  these  systems  a  target  must  not  only 
provide  realistic  IR  emissions  but  also 
replicate  the  size  and  shape  of  the  threat.  At 
present  this  can  only  met  by  means  of  full 
scale  drone  targets.  However  IR  augmentation 
of  sub  scale  targets  and  tows  might  offer  a 
cheaper  means  to  test  and  evaluate  existing 
and  future  weapons  systems  ahead  of  these 
more  expensive  trials. 

RADAR 

The  Requirement 

The  existing  aerial  Radar  targets  use  on 
DERA  Ranges  are  enhanced  using  either 
luneburg  lens  or  Semi- Active  Radar  Targets 
(SART).  The  luneburg  lens  provides  a  passive 
reflector  that  returns  a  large  percentage  of  the 
incident  energy  back  to  the  illuminating 
Radar.  The  SART  operates  slightly  differently 
in  that  it  receives  the  incident  energy  and,  by 
means  of  microwave  electronics,  amplifies  it 
before  re-transmitting.  Both  systems  give  the 
target  a  larger  Radar  Cross  Section  (RCS) 
than  it  would  otherwise  have  had  and  also  a 
larger  RCS  than  the  tug  aircraft.  However  the 
return  signal  is  just  a  larger  version  of  the 
incident  and  is  not  what  would  be  returned  by 
a  real  aerial  threat.  Real  aircraft  modify  the 
return  signal  by  introducing  effects  such  as 
scintillation,  engine  modulation  and  glint. 

Scintillation  A  complex  target  such  as  an 
aircraft  may  be  considered  as  a  number  of 
independent  scattering  elements.  The  echo 
signal  can  be  represented  as  the  vector 
addition  of  the  contributions  from  the 
individual  scatterers.  If  the  target  aspect 


changes  with  respect  to  the  illuminating  radar, 
as  might  occur  because  of  motion  of  the  target 
or  turbulence,  the  relative  phase  and 
amplitude  relationships  of  the  contributions 
from  the  individual  scatterers  would  also 
change.  Consequently,  the  vector  sum,  and 
therefore  the  amplitude  and  phase,  change 
with  changing  target  aspect. 

Glint  Changes  in  the  target  aspect  with 
respect  to  the  illuminating  radar  can  cause  the 
apparent  centre  of  radar  reflections  to  wander 
from  one  point  to  another.  In  general  the 
apparent  centre  of  reflection  might  not 
correspond  to  the  target  centre.  In  fact,  it  need 
not  be  confined  to  the  physical  extent  of  the 
target  and  may  be  off  the  target  a  significant 
fraction  of  the  time.  This  apparent  random 
wandering  of  apparent  radar  reflection  centre 
is  commonly  called  target  glint  and  results 
from  the  tracking  radar  receiving  the  vector 
sum  of  the  echoes  contributed  by  the 
individual  scattering  centres  of  the  target. 

Engine  Modulation  The  radar  echo  from 
aircraft  is  modulated  by  the  rotating  propellers 
of  piston  and  turboprop  engines.  Propeller 
Engine  Modulation  (PEM),  and  by  the 
rotating  compressor  or  turbine  blades  of  jet 
engines.  Jet  Engine  Modulation  (JEM).  A 
similar  effect  can  also  be  seen  when  looking 
at  helicopters,  the  modulation  being  caused  by 
the  rotor  blades.  Rotor  Engine  Modulation 
(REM). 

Engine  Modulation  results  in  a  number  of 
frequency  offset  returns,  the  offset  being 
relative  to  the  speed  of  the  engine,  perhaps  ten 
to  twenty  kilohertz  for  jet  engines. 

To  provide  a  realistic  radar  target  it  is 
necessary  to  emulate  these  effects. 

Solutions 

DERA  have  a  facility  based  at  DERA  Larkhill 
that  can  provide  dynamic  RCS  data  directly 
from  the  target  of  interest,  simultaneously 
measuring  amplitude  and  phase  components. 
This  dynamic  radar  signature  data  acquisition 
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system  consists  of  an  industrial  computer  with 
custom  interface  boards  and  a  phase/doppler 
interface  drawer.  This  system  interfaces  with 
a  radar  after  the  mixer-pre-IF  and  is 
transparent  to  the  host  system. 

This  system  has  been  used  to  measure  a  wide 
range  of  aircraft  over  a  wide  range  of  angles, 
on  some  aircraft  nearly  4n  steradians  (near 
spherical  coverage).  This  data  has  been  used 
to  determine  what  returns  are  required  from  a 
towed  target  to  realistically  replicate  a  threat 
aircraft. 

Semi-Active  Radar  Target  (SARTl  The 
primary  radar  towed  target  used  on  DERA 
Ranges  is  the  SART.  This  tow  is  streamed 
behind  an  Unmanned  Air  Vehicle  (UAV). 
When  activated  the  SART  receives  the 
illuminating  radar  energy  through  a  receive 
antenna  mounted  either  on  the  front,  for 
forward  looking  SART,  or  the  side  for 
sideways  looking  SART,  This  energy  is 
passed  through  a  microwave  amplifier  where 
its  power  is  greatly  increased.  This  signal  is 
then  fed  to  a  second  antenna  where  it  is  re¬ 
transmitted.  As  with  the  receive  antenna  this 
is  mounted  either  on  the  front  or  the  side 
depending  on  the  orientation  of  the  SART. 

The  SART  target  provides  a  large  RCS  return, 
however  its  output  waveform  is  identical  to 
that  falling  on  it’s  receive  antenna  but  at  a 
higher  power  level,  none  of  the  effects 
produced  by  real  aircraft  are  introduced.  This 
target  has  been  adequate  for  previous  and 
existing  generations  of  radar  guided  weapons. 
However  as  signal  processing  and  seeker  head 
technologies  advance  it  is  becoming  more 
important  to  provide  greater  realism,  both  to 
evaluate  the  weapons  performance  and  to 
protect  the  tug  aircraft. 

Modulated  Output  Semi-Active  Radar  Tow 
(MOSARTl  A  development  programme  has 
been  set-up  to  look  at  how  a  simple  SART  can 
be  modified  to  provide  a  more  realistic 
representation  of  a  threat  aircraft.  The  initial 
stages  of  this  programme  involved  the  use  of 
the  DERA  radar  signal  processor  to  assess  the 


radar  returns  from  a  selection  of  aircraft  over 
a  wide  range  of  aspect  angles.  The  data 
gathered  from  these  trials  was  analysed  and  a 
data  set  of  generic  characteristics  was 
produced. 

The  effects  that  AI  are  most  interested  in 
reproducing  are  scintillation  and  engine 
modulation,  it  was  determined  at  an  early 
stage  that  glint  would  virtually  impossible  to 
reproduce  using  the  single  radiating  source  of 
the  MOSART. 

A  feasibility  study  was  conducted  to  assess 
potential  techniques  applicable  to  the  task.  As 
a  result  it  was  decided  that  simple  modulation 
and  mixing  technologies  would  be  used.  The 
various  effects  were  then  implemented  in  the 
following  way: 

Scintillation  The  effect  of  scintillation  was 
reproduced  by  amplitude  modulating  the 
output  with  a  randomly  generated  signal  prior 
to  re-transmission.  It  should  be  possible  in 
future  iterations  to  replace  this  random  signal 
with  the  measured  characteristics  from  a  real 
aircraft.  The  phase  component  apparent  in 
true  scintillation  is  not  reproduced,  however 
future  work  will  investigate  the  possibility  of 
its  inclusion. 

Engine  modulation  The  effect  of  engine 
modulation  is  reproduced  by  the  addition  of 
frequency  offset  returns.  This  is  achieved  in 
one  of  two  ways;  The  first  involves  the 
generation  of  frequency  offsets  and  their 
addition  to  the  main  output.  Dedicated 
electronics  are  required  to  generate  each  of 
the  frequency  offsets.  This  technique  works 
well  but  requires  a  lot  of  electronics  to  truly 
reproduce  the  effects  of  engine  modulation. 
The  second  technique  uses  basic  modulating 
and  filtering  techniques  to  produce  the 
required  effects.  The  modulating  signal 
currently  used  is  simple  and  provides  a 
number  of  fixed  frequency  offsets  at  different 
amplitudes.  Future  versions  will  use  more 
complex  modulating  signals,  derived  from  the 
measured  data  and  advanced  signal  processing 
techniques,  to  replicate  the  signature  of  actual 
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aircraft,  including  rotary  aircraft. 

Glint  It  was  mentioned  earlier  that  glint  could 
not  be  produced  with  a  modified  SART  alone. 
However  some  preliminary  tests  were 
conducted  using  a  MOSART  and  a  passive 
comer  cube  reflector  separated  by  several 
metres.  By  varying  the  amplitude  of  the 
MOSART  output  it  was  possible  to  make  the 
tracking  radar  move  back  and  forth  between 
the  two  sources.  This  is  not  tme  glint  but  does 
give  a  similar  effect.  This  technique  may  be  of 
use  and  will  be  the  topic  of  a  future  work. 

Summary 

A  number  of  trials  have  been  conducted  using 
the  MOSART  both  static  in  an  anechoic 
chamber  and  in  flight  using  the  DERA  radar 
signal  processor.  Although  full  analysis  of  the 
results  has  not  yet  been  completed, 
preliminary  findings  show  the  MOSART 
radar  return  to  be  similar  to  that  of  a  real 
aircraft  and  AI  are  confident  that  within  the 
near  future  a  low  cost  realistic  radar  target 
will  be  available  for  weapons  test  and 
evaluation  trials. 


The  work  being  undertaken  by  WX3,  DERA 
Fort  Halstead  and  AI,  DERA  Aberporth  is 
aiming  to  meet  these  requirements  and  if  not 
able  to  do  away  with  the  need  for  full  scale 
targets  to  limit  the  amount  of  times  they  are 
required. 


Conclusions 

The  design  of  targets  tends  to  follow 
requirements  rather  than  be  pro-active, 
however  it  is  important  that  background 
technology  be  kept  up  to  date  to  be  ready  to 
respond  as  quickly  and  cost  effectively  as 
possible  to  new  requirements. 

Sub  scale  drone  and  towed  targets  can  offer  a 
cheap  means  of  testing  either  prototype 
countermeasures,  or  novel  seeker  systems  and 
algorithms,  and,  by  using  a  combination  of 
enhancement  and  decoys  can  be  used  to  assess 
missile  effectiveness.  Certainly  they  might 
offer  a  cheaper  means  to  test  and  evaluate 
existing  and  future  weapons  systems  ahead  of 
more  expensive  full  scale  drone  target  trials. 
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